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ABSTRACT 



A virtual city service that provides access to a virtual city 
appb'cation is provided. Users may navigate among a virtual 
representation of a real region, such as a city. The interface 
may include images of actual city blocks that have interac- 
tive items of interest such as storefronts. Users may virtually 
shop in the stores that adorn the virtual city blocks. The 
virtual city application may process orders for all mer- 
chants — local or web-based. An artificial intelligence com- 
ponent may be used in the form of a back-end linear 
programming model to provide an intelligent personal assis- 
tant. Chore-management may thus be automated. 
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METHODS AND SYSTEMS FOR PRESENTING A 
VIRTUAL REPRESENTATION OF A REAL CITY 

[0001] This application claims the benefit of U.S. Provi- 
sional application No. 60/182,282, filed Feb. 14, 2000, 
which is hereby incorporated by reference in its entirety. 

BACKGROUND OF THE INVENTION 

[0002] This invention relates to web-based virtual com- 
munities, and more particularly to an interactive, navigable, 
Web -based representation of a city. 

[0003] Virtual communities are prevalent on the Internet. 
For example, web pages that provide chat rooms, message 
forums, and e-mail exist and offer their users the ability to 
interact with one another. America Online, for example, 
offers its subscribers access to e-commerce, chat and e-mail 
features, Internet access, news, and other features. 

[0004] By offering users a sense of community while 
online, these services place the users at ease and allow them 
to be more comfortable while navigating around the virtual 
communities. However, existing virtual online communities 
do not give users a sense of realism. That is, although a user 
may be in a chat channel, perhaps directed towards conver- 
sations related to New York City, the chat channel does little 
to make the user feel like he or she is actually in New York 
City. 

[0005] Existing virtual online communities also lack the 
ability to immerse the user in the virtual community. That is, 
the user is always aware that he or she is on a Web site and 
is interacting with data. The user interfaces of existing 
virtual online communities lack the sophistication that is 
required to immerse the user in the virtual community. 

[0006] With the proliferation of the Internet, and the World 
Wide Web in particular, online shopping has grown in 
popularity. One of the major benefits of online shopping is 
the ability to access a wide range of products from a single 
web page. Because electronic merchants need only maintain 
stock in warehouses and do not have the additional overhead 
of maintaining retail stores, they are able to offer more 
products to online consumers. However, like virtual online 
communities, online shopping does not offer consumers the 
realism of real shopping. 

[0007] Rudimentary one-stop local e-commerce World 
Wide Web sites, such as Kozrao.com and UrbanFetch.com, 
arc known. For example, Kozmo.com promises to be "Ama- 
zon in under an hour." Kozmo.com warehouses books, 
compact discs, videos, and some other basic commodities 
and distributes them around select major cities via courier 
delivery. 

[0008] Kozmo.com and UrbanFetch.com (as well as other 
local e-commerce sites) are fundamentally predicated upon 
warehouse and distribution. Because the products they sell 
are so commoditized, there is little need for an interactive 
consumer experience. The only meaningful role that the 
Internet plays is that of placing an order for delivery. One 
could just as easily call in an order. Present local e-com- 
merce sites limit their ability to grow beyond staple items. 
In order to provide local e-commerce of a majority of the 
goods and services that one purchases in daily life, a very 
different approach is required. 



[0009] Another web-site, HomeDelivery.com, merely pro- 
vides links to local merchants. However, because individual 
merchants process payments, a consumer making purchases 
through HomeDelivery.com must register with and pay each 
vendor independently, effectively destroying the one-stop 
shopping model potential. 

[0010] It is therefore an object of the present invention to 
provide a virtual community that provides a sense of realism 
and immerses users in an environment. 

[0011] 1 1 is also an object of the present invention to 
provide substantially one -stop shopping to users without 
requiring the users to enter personal information multiple 
times. 

SUMMARY OF THE INVENTION 

[0012] These and other objects of the invention are accom- 
plished in accordance with the principles of the present 
invention by providing a virtual city service that gives it 
users access to a virtual city application with a comprehen- 
sive graphic user interface. The virtual city application may 
provide one-stop shopping services while not burdening 
users with repeated requests for personal information. 

[0013] The present invention is primarily described with 
respect to an embodiment in which a user interface may be 
viewed as a World Wide Web site accessible over the 
Internet. It should be understood that this is merely an 
illustrative embodiment and that any other suitable arrange- 
ment may be used. 

[0014] In one suitable embodiment of the present inven- 
tion, a user interface may be provided by the virtual city 
application of the present invention. The user interface may 
include a user-navigable virtual representation of an actual 
location, such as a city. The virtual representation may take 
the form of digitized images of the actual location that may 
be spliced together to allow navigation. For example, a user 
may navigate from one adjacent city block to another by 
selecting a horizon or any other suitable element of the 
display, such as icons. The images may be ordinary 2-D 
images, 180° images, 360° images, 3-D images, or any other 
suitable types of images. The images may be still images or 
they may be displayed as video. 

[0015] Images of city blocks may include virtual store- 
fronts. The virtual storefronts may correspond to actual 
stores in the real world. In another suitable approach, the 
storefronts may be computer-generated. Virtual storefronts 
may be treated as items of interest. Items of interest may 
generally be selectable. Upon selection, the user may be 
directed to a web page (in the case of a web -based imple- 
mentation) that provides more information on the item of 
interest. For example, upon selection of a storefront, the user 
may be directed to a virtual representation of the interior of 
the store in which the user may navigate and view actual 
items for sale as though he or she were in a real store. 

[0016] The virtual city application may also provide links 
(e.g., as an item of interest in an image or as an ordinary 
hyperlink) to web-based merchants. Whether large web- 
based merchants or small local merchants, the virtual city 
application may provide users with the convenience of 
one-stop shopping. For example, purchases made with mer- 
chants that are partnered with the virtual city service may be 
processed by the virtual city application rather than by the 
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individual merchants. In one suitable approach, the virtual 
city application may request that the user provide personal 
and payment information (al registration, for example). This 
information may subsequently be used for all purchases 
made through the virtual city application without having the 
user repeatedly provide the information to numerous mer- 
chants. The virtual city application may also provide a single 
virtual shopping carl, into which users may place items they 
wish to buy from among any of the merchants that are 
partnered with the virtual city service. The virtual city 
service may take a commission for any order that it pro- 
cesses. 

[0017] The virtual city application may also act as a data 
gatherer. In one suitable approach, the virtual city applica- 
tion may track the patterns of use by its users. These patterns 
may relate to shopping, which virtual regions are visited, or 
to any other suitable criteria. The data may be used by 
merchants and by the virtual city service for marketing 
purposes or for any other suitable purpose. 

[0018] Advertisements may be displayed as embedded 
advertisements within virtual city images. For example, 
cars, clothes, signs, banners, billboards, or any other suitable 
elements may all be selectable in order to provide more 
information about the manufacturer, product, etc. In one 
suitable approach, embedded advertisements may be 
inserted into the virtual city images using graphics process- 
ing. That is, the embedded advertisements may exist in the 
virtual world, but not in the real world. 

[0019] The virtual city application may also provide chore 
management utilities. For example, plumbers, doctors, law- 
yers, dentists, mechanics, or any other service professionals 
may be searched for and contacted (e.g., for an appoint- 
ment). 

[0020] The virtual city application may also provide vir- 
tual travel and tourism features. For example, interest- 
specific virtual tours may be provided whereby users may 
take virtual tours based on particular preferences (e.g., jazz 
in New Orleans). The virtual tours may include visits to 
parades, museums, clubs, restaurants, or any other suitable 
event or location. 1 Tie user may navigate among the events 
or locations using a process similar to that for navigating 
among virtual city blocks. 

[0021] The virtual city application may also include an 
artificial intelligence component. For example, an intelligent 
personal assistant may be implemented that may provide 
automated chore-management. Decisions may be made by 
the intelligent personal assistant based on a linear program- 
ming model having a large number of variables. With regard 
to chore-management, some of these variables may include 
the user's location, personal preferences, habits, schedule, 
income, or any other suitable variables. The personal assis- 
tant is merely one example of how a linear programming 
model may be used with the present invention. Any other 
suitable implementation of a linear programming model may 
be made. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0022] FIG. 1A is a block diagram of an illustrative 
image-capturing system in accordance with an embodiment 
of the present invention. 



[0023] FIG. IB is a flow chart of illustrative steps 
involved in making images available to the virtual city 
application in accordance with an embodiment of the present 
invention. 

[0024] FIG. 2 is a block diagram of an illustrative virtual 
city image distribution system in accordance with an 
embodiment of the present invention. 

[0025] FIG. 3 is a block diagram of an illustrative infor- 
mation distribution system in accordance with an embodi- 
ment of the present invention. 

[0026] FIG. 4 shows an illustrative virtual city application 
display screen for navigating around a virtual city in accor- 
dance with an embodiment of the present invention. 

[0027] FIG. 5 shows the illustrative virtual city applica- 
tion display screen of FIG. 4 in full mode in accordance with 
an embodiment of the present invention. 

[0028] FIG. 6 is a flow chart of illustrative steps involved 
in using a map to navigate around a virtual city in accor- 
dance with an embodiment of the present invention. 

[0029] FIG. 7 is a flow chart of illustrative virtual city 
application display screens for navigating around a virtual 
city block image in accordance with an embodiment of the 
present invention. 

[0030] FIG. 8 shows an illustrative virtual city application 
display screen for a store information page in accordance 
with an embodiment of the present invention, 

[0031] FIG. 9 is a block diagram of an illustrative arrange- 
ment of features that may be offered by the virtual city 
application in accordance with an embodiment of the present 
invention. 

[0032] FIG. 10 shows an illustrative virtual city applica- 
tion display screen for using embedded advertisements in 
virtual city images in accordance with an embodiment of the 
present invention. 

[0033] FIG. 11 is a flow chart of illustrative steps involved 
in navigating among virtual city blocks in accordance with 
an embodiment of the present invention. 

[0034] FIG. 12 shows an illustrative virtual city applica- 
tion display screen for seamlessly integrating the virtual city 
interface with other web pages in accordance with an 
embodiment of the present invention. 

[0035] FIG. 13 is a flow chart of illustrative steps involved 
in using the virtual city application to process orders in 
accordance with an embodiment of the present invention. 

[0036] FIG. 14 is a block diagram of an illustrative 
arrangement for sharing customer data and marketing infor- 
mation between the virtual city application and merchants in 
accordance with an embodiment of the present invention. 

[0037] FIG. 15 is a block diagram of an illustrative 
arrangement of features that may be offered by the virtual 
city application in accordance with an embodiment of the 
present invention. 

[0038] FIG. 16 shows an illustrative virtual city applica- 
tion display screen for using an avatar in accordance with an 
embodiment of the present invention. 
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[0039] FIG. 17 is a block diagram of an illustrative 
relationship between an intelligent personal assistant and a 
linear programming model in accordance with an embodi- 
ment of the present invention. 

[0040] FIG. 18 is a block diagram of an illustrative 
arrangement of revenue sources provided by the virtual city 
application in accordance with an embodiment of the present 
invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

[0041] The present invention relates to a localized navi- 
gation system that may use digital imagery and sound to 
recreate, in exacting detail, a virtual reality simulacrum of 
any desired real world city. 

[0042] In the preferred embodiments of the present inven- 
tion, every virtual city may be a digital replica of a real- 
world city in which users in that community and around the 
world are allowed to explore, browse, and mingle. Although 
it may be possible to simulate a real-world city with a virtual 
city using substantial precision, it may not always be desir- 
able to do so. For example, in order to maintain an envi- 
ronment that is directed towards the general public, offen- 
sive material may be filtered out. 

[0043] In one suitable embodiment, the present invention 
may be implemented as a World Wide Web site that users 
may access via, for example, the Internet. As it currently 
stands, the Internet is a "dumb" finder and sorter of infor- 
mation. The present invention may give the Internet the 
potential to provide a "smart" sensory experience. 

[0044] For purposes of brevity and clarity, the present 
invention is described as a virtual city service made avail- 
able to the public as a virtual city application via the Internet. 
It should be understood that this is merely an illustrative 
embodiment. Any other suitable hardware, software, proto- 
cols, networks, or arrangements may be used. 

[0045] The present invention may provide a new type of 
Internet service. Both business-to-business and business-to- 
consumer, customers of the virtual cities service may be 
varied and many. Large conglomerates and family-owned 
businesses may both have their place in the virtual city 
service's online community. Individual users with interests 
as diverse as traveling, shopping, computer-gaming, and 
simplifying the business of their daily lives may come to rely 
on the virtual city service as their first and most-trusted 
venue on the web. Large vendors may make use of the 
virtual city service to generate incremental sales and to 
provide their customers far greater ease of checkout and 
payment — without losing control of the customer payment 
information. Small to medium-sized vendors may make use 
of the virtual cities service to make a presence on the web 
as part of a large, robust, dynamic community within which 
they will enjoy economies of scale and access to unique and 
valuable marketing data. 

[0046] FIG. 1A shows an illustrative system that may be 
used for procuring images to be used by a virtual city 
application. The term "image," as used herein, may refer to 
either still images, a slide show of still images, video, a slide 
show of videos, animation, or any other suitable form of 
imagery. The basic unit of a virtual city may be city block 
100. City block 100 may be any suitable fundamental 



portion of a city. For example, city block 100 may be an 
intersection, a section of a street, court, or avenue, a section 
of a road, an area having particular dimensions (e.g., in the 
shape of a square, rectangle, circle, etc.), or any other 
suitable fundamental portion of a city. Using any suitable 
image capturing device 102, such as a 180° digital camera, 
360° digital camera, ordinary digital camera, film-based 
camera, video camera, or any other suitable image capturing 
device, images of city blocks may be captured. 

[0047] Some or all of the captured images may be edited, 
stitched, or manipulated in any other suitable way using, for 
example, graphics editing equipment 101. Graphics editing 
equipment 101 may be any suitable equipment, such as 
personal computing equipment, super-computing equip- 
ment, silicon graphics computing equipment, any other 
suitable equipment, or any combination thereof. The images 
may be stored in image database 110, which may be located 
at server 103. 

[0048] In one suitable embodiment of the present inven- 
tion, images captured by image capturing device 102 need 
not be in a digitized format. For example, the images may be 
hard-copy photographs. At step 104 of FIG. 104, it may be 
determined whether the images are digitized images. If the 
images are not digitized, then the images may become 
digitized at step 106 using, for example, scanning equipment 
and suitable software. In the case of video, frame capture 
hardware and software may be used at step 106 assuming a 
still image embodiment of the virtual city application is 
being used. If virtual city blocks can be displayed using 
video, then video clips, streaming video, or any other 
suitable form of digitized video may be used. 
[0049] The digitized images may then be edited and 
stitched together using any suitable stitching technology at 
step 105 and stored in database 110 at step 107. The stitching 
may link the images of the city blocks to one another in 
image database 110. Whatever links exist in the real city, 
such as from one city block to another, the same links may 
exist between the digital images of the virtual city. The result 
may be a network of inter-linking images that may be freely 
roamed, from one image to another, much like real life. This 
is merely an illustrative arrangement of hardware and pro- 
cesses. Any other suitable arrangement and processes may 
be used. 

[0050] FIG. 2 shows an illustrative system for the present 
invention. Image data 112 may be loaded into data server 
114 that holds image database 116. Image database 116 may 
be accessed by user hardware 122 using communications 
network 120. Communications network 120 may be the 
Internet, a modem connection, a wireless communications 
path (e.g., using wireless access protocol (WAP)), a satellite 
connection, a television cable connection or any other suit- 
able peer-to-peer based or client-server based communica- 
tions network. Data server 114 is shown to be coupled to 
communications network 120 using path 118. Path 118 may 
include intervening hardware, such as one or more web 
servers, data servers, proxy servers, gateway servers or any 
other suitable intervening hardware. If desired, these or any 
other suitable servers may be combined so that less hard- 
ware is used. For example, data server 114 may be combined 
with a web server. Any such suitable combination may be 
used if desired. 

[0051] It should be understood that although only image 
data 112 is shown in FIG. 2, any other type of data may be 
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acquired and stored in a database for use by the virtual cities 
application. For example, audio data, text data, advertise- 
ment data, navigation data, or any other suitable data may be 
stored in data server 114 or in any other suitable server. Any 
or all types of data may be stored in a single data server. If 
desired, multiple data servers may be used to store data. Data 
servers may be separate servers dedicated to holding one or 
more databases, or alternatively, data servers may be com- 
bined with other types of servers, such as web servers, proxy 
servers, or any other suitable type of server. 

[0052] User hardware 122 may be any suitable user hard- 
ware, software, or both. For example, user hardware 122 
may be a personal computer with web-browsing software 
that may be used by users of the virtual city application to 
navigate among a virtual city via communications network 
120. Alternatively, user hardware 122 may be a personal 
computer with image-capturing hardware and suitable 
image-processing software that is located at a local retail site 
(e.g., a grocery store). In this case, user hardware 122 may 
provide data server 114 with updated images of the local 
retail site via communications network 120. In another 
suitable approach, updated images of the local retail site may 
be made available to data server 114 using any other suitable 
arrangement. 

[0053] FIG. 3 shows an illustrative back-end system that 
may be used for the present invention. Acentral site 113 may 
be used to hold data server 114 and information database 
115. Central site 113 may be in one physical location, or, 
alternatively, central site 113 may include components that 
are located at more than one location. 

[0054] Information database 115 may be stored in data 
server 114, or, alternatively, information database 115 may 
be stored on a separate server. If desired, information 
database 115 may be partially stored in data server and 
partially stored on one or more other servers. Information 
database 115 may be used to store user information such as 
names, addresses, telephone numbers, and any other suitable 
user information. In one suitable approach, user information 
may include any suitable payment information such as credit 
card information, bank account information, or any other 
suitable payment information. In yet another suitable 
approach, user information may include behavioral data. 
Behavioral data may be acquired for each user based on the 
user's usage patterns in the virtual city application, such as 
which local merchants the user often visits using the virtual 
city application or any other such suitable usage patterns. 

[0055] Remote sites 119 may access information database 
115 using communications network 117. Communications 
network 117 may be the same as communications network 
120, or communications network 117 may otherwise be any 
such suitable communications network. Remote sites 119 
may include local merchants, web-based merchants, credit 
processing stations, marketing consultants, or any other 
suitable remote sites. 

[0056] One of the features of the present invention is the 
interface of the virtual city application. In a real world city, 
any given block might contain many items of interest, such 
as storefronts, monuments, parks, statues, famous buildings, 
or any other suitable items of interest. The virtual city image 
of that block may contain one or more links (e.g., hyper- 
links), each associated with the corresponding real world 
item of interest. In a web-based approach, hyperlinks maybe 



embedded within the image of the city block. If desired, the 
hyperlinks may be made available to the user outside of the 
image of the city block, such as in a separate listing of 
hyperlinks associated with the currently displayed city 
block. This separate listing may be displayed in a separate 
section on the user display screen. In another suitable 
approach both a separate listing of hyperlinks and embedded 
hyperlinks may be used. When the user selects a hyperlink, 
additional information regarding the item of interest asso- 
ciated with the selected hyperlink may be displayed to the 
user. 

[0057] FIG. 4 shows an illustrative virtual city application 
display screen 124. City block image 126 may be displayed 
to the user. City block image 126 may include storefronts 
such as storefront 128. The user may position a cursor 130 
over storefront 128 and may select storefront 130 as a 
hyperlink. In addition, hyperlink listings 132 may be dis- 
played on virtual city application display screen 124, allow- 
ing the user to select an item of interest from a textual based 
list rather than directly from the image. For example, listing 
134 corresponds to storefront 128. Either the embedded 
hyperlink or the text-based listing may be selected by the 
user to display additional information about that particular 
item of interest. 

[0058] The hyperlink listing feature is particularly useful 
when the item of interest in not visible or is poorly visible 
in the city block image. For example, city block image 126 
corresponds to a New York City block on which the "Van- 
guard Delicatessen" is found. However, the Vanguard Deli- 
catessen storefront is not viewable in city block image 126. 
The user may retrieve more information about the Vanguard 
Delicatessen via its corresponding hyperlink listing 136. 

[0059] If desired, hyperlink listings 132 may also include 
hyperlinks to items of interest that are related to items of 
interest found in the corresponding city block. For example, 
if a car dealership is a particular item of interest on a 
particular city block, then a possible additional hyperlink 
listing may be a hyperlink to a car mechanic in the imme- 
diate area. Any such suitable use of hyperlink listing 132 
may be used. 

[0060] A map such as map 138 may be displayed on 
virtual city display screen 124. Map 138 may show the user 
the location of the presently displayed city block 126. If 
desired, map 138 may also show neighboring attractions, 
such as parks, monuments, etc. Hotels, theaters, and nearby 
restaurants may also be displayed on map 138. This is 
merely an illustrative use of maps in the virtual city appli- 
cation. Any other suitable arrangement may be used. 

[0061] Advertisements, such as advertisement 140, may 
be displayed on virtual city display screen 124. Advertise- 
ment 140 may be a banner advertisement, or if desired, 
advertisement 140 may be displayed in the form of panel 
advertisements. Advertisement 140 may contain text, graph- 
ics, video (including animation), or a combination of these 
or other suitable types of content. Advertisements may be 
interactive. For example, advertisement 140 has interactive 
regions 141 and 143. If the user selects interactive region 
141 (e.g., using cursor 130), a web page corresponding to 
Apartment.com may be opened and displayed. Likewise, if 
interactive region 143 is selected, a web page corresponding 
to an apartment database at Apartment.com may be opened 
and displayed. Advertisements need not contain multiple 
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interactive regions, but may simply be selectable. If desired, 
passive advertisements may be used. Any suitable adver- 
tisements may be used in conjunction with the virtual city 
application. Advertisement 140 is merely an illustration. 

[0062] Updated information such as news, local events, 
weather, reviews, etc. may be displayed on virtual city 
application display screen 124. For example, city events and 
information 148 may be displayed. Local news 146 may be 
displayed. Local weather 144 may be displayed. The 
updated information may correspond to the city in which the 
presently displayed city block is. If desired, updated infor- 
mation regarding the neighborhood in which the presently 
displayed city block is may be displayed. If desired, updated 
information regarding the block itself may be displayed. The 
information displayed on virtual city application display 
screen 124 may include snippets of information that may be 
made available to the user upon request. For example, local 
news 146 may include short headlines to make the user 
aware of the top local news stories. For additional, more 
in-depth coverage of the local news, the user may select a 
hyperlink associated with local news 146 (e.g., a particular 
headline may be selected, or local news title 147 may be 
selected, etc.). Once selected, a new web page may be 
displayed having additional information. This is an illustra- 
tive arrangement of displaying local information on a virtual 
city application display screen. Any other type of suitable 
arrangement may be used. 

[0063] In one suitable embodiment, the virtual city appli- 
cation may offer the user an opportunity to search for items 
of interest. For example, one or more keywords may be 
entered by the user in search box 142. The search may be 
limited to searches by category (as illustrated), or the 
searches may be limited to other parameters. In another 
suitable approach, no limitations need be placed on searches. 

[0064] Various items on the virtual city application display 
screen may be shown in more than one mode. For example, 
city block image 126 may be shown in either full screen 
mode or in regular screen-size mode. The default mode may 
be the regular mode. Screen 124 shows city block image 126 
in regular mode. A full option 150 may be provided that may 
allow the user to enlarge city block image 126 so that it fills 
the entire browser window or the computer screen. In full 
mode, every other item of functionality may become a 
button at the top or bottom of the display screen (or browser 
window, if the browser window is not maximized). This is 
illustrated in FIG. 5. City block image 158 is the full mode 
version of city block image 126 (FIG. 4). Buttons 160 may 
correspond to the various items of functionality displayed in 
screen 124 of FIG. 4. For example, map button 162 in screen 
156 may correspond to map image 158 in screen 124. To 
retrieve the image of the map, the user may select map 
button 162. This may cause a full mode map image to 
appear. In one suitable approach, selecting map button 162 
may cause the display screen to be changed to regular mode, 
thus displaying the map image in regular mode. If desired, 
selecting map button 162 may cause a new web page to be 
displayed that includes a map image. The new web page may 
be displayed in place of the currently displayed web page in 
the current browser window, or the new web page may be 
displayed in a new browser window overlaying the current 
browser window. 

[0065] In one suitable embodiment, resolution options 149 
and 151 may be provided to the user. Because the connection 



speed of different users may stretch across a large range, the 
user may be given the opportunity to choose a resolution 
level of the virtual city block images that best corresponds 
to the connection speed being used by the user. The reso- 
lution level may include the level of image quality, image 
size, colors used, or any other suitable components. 

[0066] Density options 145 and 147 may be provided to 
the user The user may be given the opportunity to choose a 
density level of items of interest and virtual city blocks to be 
displayed. The density level may correspond to whether 
certain virtual city blocks should be displayed based on the 
number of items of interest that are present on the blocks. 
For example, if the user is using a slow modem connection, 
then he or she would not wish to download images of virtual 
city blocks that are empty alleys. By setting the density to a 
lower level, these virtual city blocks may be skipped as the 
user navigates through the virtual city. 

[0067] Density options 145 and 147 and resolution options 
149 and 151 are merely illustrative. If desired, more choices 
of resolution levels and density levels may be provided (e.g., 
a quantitative resolution level may be used). If desired, the 
resolution level and density level may be automatically 
assigned by the virtual city application that may take into 
account the user's connection speed and personal prefer- 
ences. The user may still, however, retain the ability to alter 
these automatic settings. 

[0068] One of the navigation utilities that the virtual city 
application may provide to the user is the ability to select a 
point or a region in map 138 to view the corresponding 
virtual city block image for the selected point or region. This 
process is illustrated in FIG. 6. At step 164 a map is 
displayed on the virtual city application display screen. The 
user may choose a point or region on the map, using, for 
example, a cursor controlled by a mouse. In response, at step 
166, the virtual city application may display a virtual city 
block image that may correspond to the point or region 
selected by the user. 

[0069] The full mode option may also be provided for map 
138 to provide greater detail. This is helpful, for example, in 
order to view the map in block-by-block detail. Furthermore, 
this feature may provide greater ease of navigation. Whereas 
selecting an area on the regular mode map may direct the 
user to the general area selected on the map, clicking on the 
full mode map may direct the user to the exact block the user 
desires. This is merely an illustrative use of the map feature 
of the present invention. Any other suitable use may be 
implemented for the map feature. 

[0070] Audio option 152 may also be provided. Audio 
option 152 may allow the user to listen to location-specific 
ambient sound. Each virtual city block may be associated 
with a code (e.g., stored in a relational database) that 
corresponds to an audio track. These audio tracks may 
include sounds typically heard on highways, busy streets, 
residential streets, intersections, parks, etc. The user may 
activate the audio feature by selecting audio option 152. 
Likewise, the user may be given the ability to deactivate the 
audio feature by selecting an appropriate item on the virtual 
city application display screen. Any such suitable use of an 
audio feature may be used. If desired, other, more interactive 
sounds may be provided (e.g., a microphone may be 
installed on all or particular city blocks, and the user may be 
given the opportunity to listen to the sounds in real-time). 
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Alternatively, the sounds of a particular block may be 
recorded as audio clips available to the user at a later time. 

[0071] Virtual city block image 126 (as well as the coun- 
terpart full mode virtual city block image 158) need not be 
static. The virtual city application may provide the user with 
the ability to navigate within the virtual city block image. 
For example, the user may be given the ability to use a 
mouse (or the cursor keys on a keyboard) to move within the 
virtual city block image. This feature is particularly useful 
when the virtual city block image is a 180° image or a 360° 
image because the user may be given the ability to navigate 
the city block to a higher degree using the virtual city block 
image. The navigation may be accomplished by clicking on 
the virtual city block image and holding down either the left 
or right mouse button. This may alter the perspective and 
create the appearance of movement. Navigation functional- 
ity may include pan left, pan right, pan up, pan down, zoom 
in, zoom out, rotate, any combination thereof, or any other 
suitable function. FIG. 7 illustrates the use of some of these 
navigation functions. Virtual city block image 168 may be a 
360° image of a particular real city block. If the user zooms 
in and pans up, virtual city block image 170 may be 
displayed. If the user then issues a command associated with 
counterclockwise rotation, virtual city block image 172 may 
be displayed. The navigation may take place in a smooth and 
continuous manner, giving the user the impression of actual 
movement. That is, the user may feel that he or she is 
actually moving his or her head to get a different view of the 
city block. 

[0072] In one suitable embodiment, selecting a storefront 
such as storefront 128 may link the user to a web page 
containing a detailed image of that establishment, with 
information on its location, phone numbers), nearest sub- 
way stop, and the goods sold there. If desired, any other 
suitable information may be provided. An illustrative infor- 
mation web page 174 is shown in FIG. 8, A list 176 of the 
most popular items or categories of items sold at the selected 
store may be displayed to the user. Any other suitable 
number of most popular items may be displayed. If desired, 
other criteria may be used for displaying items. For example, 
items may be displayed based on the merchant's choice 
based on, for example, current sale items. Any such suitable 
criteria may be used to determine which items to display. 
This feature may allow the user to have an accurate sense of 
whether the shop carries the particular item for which he or 
she is looking. Other information 178 about the store may be 
provided, such as its history or a brief biography of the 
proprietors. In one suitable approach, the navigable virtual 
image feature as hereinbefore described for virtual city 
block images may be used for the interiors of stores and 
other establishments selected by the user. Virtual shop image 
180 may therefore be displayed on web page 174. The user 
may be given the ability to navigate around the virtual store 
image (e.g., using panning, zooming, and rotating features) 
using, for example, either regular mode or full mode. In one 
suitable approach, users may navigate around a virtual store 
using any suitable features, including, but not limited to, 
those shown in FIGS. 4 and 5. The preceding is merely an 
illustrative embodiment of a store information web page. 
Any other suitable design, arrangement, information, and 
features may be used. 

[0073] Every storefront in every virtual city block image 
may be selectable by the user. In another suitable approach, 



some storefronts may merely be passive images. Upon 
selection of a storefront, a web page such as web page 174 
may be displayed to provide further information about the 
selected storefront. In another suitable approach, any other 
suitable web page may be displayed in response to a 
selection of a storefront. 

[0074] This is merely an illustrative use of information 
web pages. Similar detail may be made available for any or 
all items of interest — not just storefronts. Every park and 
monument, every restaurant and theater, every church and 
community center, may have corresponding information on 
its history, hours of operation, show-times, reviews, and 
goods and services provided, or any other suitable informa- 
tion made available via the virtual city application. 

[0075] In one suitable embodiment of the virtual city 
service of the present invention, information web pages such 
as web page 174 may be provided to merchants (i.e., 
merchants of the storefronts in the virtual city blocks) 
without cost. This will allow for a greater number of stores 
to be represented in the virtual city. 

[0076] In another suitable approach, web-based e-com- 
merce merchants may have a presence in the virtual city 
community as well. For example, virtual storefronts may be 
created in place of, for example, empty lots. These virtual 
storefronts may be used by, for example, e -commerce mer- 
chants that do not have any physical establishments in the 
real city. In another approach, real stores may change their 
locations or add new locations by creating virtual storefronts 
in any suitable location in the virtual city. 

[0077] In one suitable embodiment of the present inven- 
tion, the virtual city application may be used to display 
advertisements to users. FIG. 9 shows advertisements 196 
being used in the virtual city application. Advertisements 
196 may include banners, icons, text, video, images, embed- 
ded advertisements, and any other form of advertisements. 
Embedded advertisements are advertisements that are part of 
the virtual city block images. For example, FIG. 10 shows 
a typical view of a virtual city block image. Sign 206 is an 
advertisement for McDonald's. The virtual city application 
may allow the user to select sign 206 as an embedded 
advertisement. Upon selection, the user may be directed to 
a web page giving more information about the advertised 
product or service. This feature may be applied to billboards, 
posters, signs, murals, clothing, automobiles, and essentially 
anything visible in the virtual city image. 

[0078] If an advertiser, retailer, manufacturer, or merchant 
does not wish to have real advertisements and items that are 
visible in the virtual city block image to be embedded 
advertisements, those items may be made to be passive. 
Alternatively, those items may be substituted with other 
items for which another advertiser, retailer, manufacturer, or 
merchant does want an embedded advertisement. This may 
be done using any suitable computer graphics software to 
edit the virtual city block image. 

[0079] In one suitable approach, embedded advertise- 
ments may be used by local merchants to the extent that real 
world advertisements can not. For example, a local grocery 
store may advertise itself using, for example, an embedded 
advertisement such as a billboard that may be added to a 
virtual city block image using any suitable graphics pro- 
cessing hardware, software, or both. This billboard may be 
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viewed by residents of the neighborhood, visitors of the 
neighborhood, or by any other users of the virtual city 
application by navigating to the virtual city block. In this 
respect, smaller, local merchants may reach the large audi- 
ences a billboard is intended to reach, but without going 
through the process (e.g., acquiring permits, payment of 
large fees, etc.) of physically placing a billboard up. This is 
merely an illustrative approach to local advertising. Any 
other suitable approaches may be used. 

[0080] In one embodiment of the present invention, the 
virtual city application may allow users to navigate around 
the virtual city by providing selectable horizons. That is, the 
user may select the end of the currently displayed city block 
represented by the virtual city block image (e.g., the end of 
the street). This may cause a virtual city block image 
corresponding to the block adjacent to the currently dis- 
played virtual city block to be displayed. The direction may 
be determined by the area the user selected. For example, if 
the user selects the end of the block, the next block after the 
point selected may be displayed. If the user selects the other 
end of the block, then the block adjacent to that end may be 
displayed. Likewise, if the user selects a side of the dis- 
played block, the block adjacent to the selected side may be 
displayed. This method of navigation is illustrated in FIG. 
11. 

[0081] In addition to user interface features, the virtual 
city application of the present invention may offer its users 
many types of on-line resources using the interactive nature 
of its interface. Some of these resources, as illustrated by 
FIG. 9, may include, but are not limited to, links to general 
e -commerce web sites 188 (e.g., Amazon.com, Buy.com, 
etc.), links to other local e-commerce sites 186 (e.g., Urban- 
Fetch. com., Kozmo.com, etc.), links to service providers 
184 (e.g., on-line banking, etc.), links to existing on-line city 
guides 190 (e.g., sidewalk.com, etc.), links to existing on- 
line restaurant guides 192 (e.g., Zagats.com, etc.), links to 
existing on-line local newspapers 194 (e.g., nypost.com, 
etc.), links to existing on-line national newspapers (e.g., 
msnbc.com, etc.), as well as links to any other suitable 
on-line resources. Agreements between the provider or pro- 
viders of the virtual city application and the providers of the 
on-line resources being linked from within the virtual city 
application may be made. 

[0082] If desired, certain modifications may be made to 
the on-line resource web page so that the user may be 
directed to the on-line resource web page from the virtual 
city application seamlessly. For example, these modifica- 
tions may include having the color scheme of the on-line 
resource match the color scheme of the virtual city appli- 
cation interface. If desired, these modifications may include 
displaying the on-line resource web page in a frame within 
the virtual city application interface. If desired, an interface 
such as that illustrated in FIG. 12 may be used, whereby the 
on-line resource web page may be displayed in combination 
with virtual city application content without the use of 
frames. This method of seamless integration of content 
provides a co-domain environment in one web page. Head- 
ing 200 may be displayed and may correspond to the 
provider of the virtual city application. Heading 200 may be 
selectable. If the user selects header 200, the user may be 
redirected to the virtual city web page. 

[0083] While navigating in an e-commerce web site, or 
any other type of web site, the user may always return to the 



main virtual city application web site. For example, after 
making a purchase using the virtual city application to 
process the order, the user may be returned to the virtual city 
application web site. Even if the user does not purchase 
anything, the user may always be given the ability to return 
to the virtual city application web site. For example, in FIG. 
12, the user may simply click on heading 200 (which may 
be displayed on every web page of the web site, no matter 
how deeply into that web site the user travels) and be 
returned to the virtual city application web site. Any other 
suitable method, interface design, or both may be used for 
always giving the user the ability to return to the virtual city 
application web site. 

[0084] In the case of e-commerce web sites, such as the 
one shown in FIG. 12, if the user desires to make a purchase, 
and selects the appropriate icon, link, banner, button, etc., 
the user may process his or her purchase order through the 
virtual city application and the virtual city service. For 
example, if the user wishes to purchase an item from 
Gap.com, the user may progress through the Gap.com web 
page (that may have been seamlessly integrated with the 
virtual city application and the user may view the content 
normally displayed on Gap.com, including product descrip- 
tions, item descriptions and images, etc. When the user 
wishes to make a purchase, however, the order may be 
processed through the virtual city application rather than 
through Gap.com's own servers. Payment may be made to 
the virtual city service rather than to Gap.com. This process 
is illustrated in FIG. 13, which show customer orders 204 
being processed by virtual city application 205, that in turn 
sends orders and stored customer data 208 to appropriate 
merchants 207 for fulfillment. Orders 204 may include any 
suitable information about customer orders, such as item 
descriptions, item identification numbers or codes, quantity 
information, color and size information, or any other suit- 
able customer order information. Order and customer data 
208 may include the customer order information of order 
204 as well as any suitable customer data that is stored by 
the virtual cities application. 

[0085] By linking all order processes from all e-commerce 
web sites accessible through the virtual city application, the 
user is given the ability to shop for items at multiple places, 
each purchase being made through a single source. This 
allows the user to register only once. The subjection of the 
user to queries for personal and payment information is 
therefore minimized using the virtual city application. 
Instead of having to register and provide payment informa- 
tion at every web site at which the user wishes to make a 
purchase, the user may simply have all purchases made 
through the virtual city service using the virtual city appli- 
cation. In one suitable approach, the user may also be given 
the ability to go from web site to web site, adding items to 
a virtual shopping cart. After all desired items are in the 
virtual shopping cart, the user may process the order using 
the virtual city application. 

[0086] The one-stop method of e-commerce shopping as 
described is merely illustrative. Any other suitable method 
may be used. For example, the virtual city application need 
not process any orders. If desired, the virtual city application 
may only process orders for certain merchants, perhaps 
based on whether the merchant desires to take advantage of 
the feature. 
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[0087] FIG. 13 also shows customer data being trans- 
ferred from the virtual city application to the e-commerce 
site. Customer data may include information such as infor- 
mation that the user entered when the user registered for the 
virtual city service. This may include such information as 
name, addresses, telephone numbers, e-mail addresses, sur- 
vey answers, as well as any other suitable information. The 
e-merchant may be provided with the information without 
the user having to enter the information again. The infor- 
mation may be used for any suitable purpose. For example, 
the information may be used for marketing purposes, for 
building a customer database, for delivery information, or 
for any other suitable purpose. 

[0088] The virtual city application may also provide mer- 
chants (including large e-commerce merchants as well as 
local shops) with marketing data for particular regions. The 
virtual city application may compile marketing data based 
on the sales it processes. In an alternative approach, mar- 
keting data may be based on any other suitable interactions 
between users and the virtual city application. This data may 
be provided to merchants in order to, for example, better 
serve the merchants' marketing efforts in terms of, for 
example, advertising. FIG. 14 shows merchants receiving 
individual customer information as well as general market- 
ing data for one or more regions of a city. Virtual city 
application 210 may provide customer information 211 and 
marketing data 212 to merchant 213. The precision of the 
marketing data is high given the fact that the virtual city 
application is collecting a vast amount of data by acting as 
the middleman in e-commerce transactions that spread 
across a wide spectrum of merchants. 

[0089] The virtual city application may be made to be as 
robust as desired. FIG. 15 illustrates some other features 
that the virtual city application may offers its users. These 
features may include chore management services, apartment 
searching services, bulletin board messaging and chat ser- 
vices, an avatar feature, travel services, local television 
programming, and an intelligent information assistant. The 
next several paragraphs will describe these features in more 
detail. It should be understood that any such suitable features 
may be implemented and that these features, herein 
described, are merely illustrative. 

[0090] Chore management services may allow the user to 
access services (i.e., as compared to products) using the 
virtual city application. These services may include, but are 
certainly not limited to, allowing the user to pay bills online, 
schedule appointments (e.g., for doctors, lawyers, job inter- 
views, plumbers, etc.), make restaurant reservations, do 
on-line banking, balance a checkbook, as well as any other 
suitable services, including giving the user the ability to find 
and try new service providers. As with product purchases, 
service purchases may be made using the virtual city appli- 
cation as the payment processor in accordance with the 
present invention. 

[0091] The virtual city application may also provide apart- 
ment searching services. For example, real estate brokers 
and agents may have listings of available apartments (and 
houses) available through the virtual city application. The 
user may select these listing to be given a virtual tour of the 
apartment. The virtual tour may be given in a manner similar 
to allowing the user to navigate among virtual city blocks as 
described hereinbefore. That is, the user may be given the 
ability to navigate around the apartments using, for example, 
360° images of the apartment. The user may be given the 
ability to open doors, look in closets, look out the window, 



and hear the sounds in the apartment (e.g., to find out how 
noisy it is). If desired, a virtual real estate agent may speak 
as the user lours the apartment. The pre-recorded speech 
may correspond to what the user is currently viewing. The 
user may be given the ability to navigate around the neigh- 
borhood by simply using the virtual cities application to 
view the virtual city blocks as hereinbefore described. This 
is merely an illustrative description of the apartment search- 
ing feature. It should be understood that any other suitable 
method of using the present invention for apartment search- 
ing may be used. For example, private owners may be given 
the ability to place links in a classifieds section of the virtual 
city application, advertising their apartment for rent. It 
should also be understood that this feature may be expanded 
to include apartment sales, house sales, land sales, commer- 
cial building leasing and sales, etc. 

[0092] Because the virtual city application is intended to 
represent a real community, bulletin board messaging and 
chat functionality may be added. Users may post messages 
or chat about certain topics or may post messages and chat 
to each other based on where in the virtual city they 
currently are. For example, users viewing the virtual South 
Street Seaport in Manhattan may be given the ability to chat 
to other users currently viewing the South Street Seaport as 
well. 

[0093] Avatars may also be provided by the virtual city 
application. Avatars may be graphical representations of 
online identities. That is, an image of a person or any other 
suitable entity may represent a real person navigating around 
virtual city. These images may be actual images of the users 
using the virtual city application, they may be arbitrary 
images, or they may be any images chosen by the users. 
Avatars may allow people from distant places to meet at a 
particular place in the virtual city environment and, for 
example, go shopping at the mall. Interviews may be con- 
ducted in this fashion. Meetings may be held in a virtual 
conference between attendees that are hundreds or thou- 
sands of miles apart. FIG. 16 illustrates how avatars may be 
displayed to the user using the virtual city application. 
Avatar 220 may represent one of the users, while avatar 222 
may represent another user. The two users represented by the 
avatars may interact with one another using chat functions or 
messaging functions. In one suitable approach, the users 
may use microphones to transmit their voices to one another. 
The preceding description is merely an illustrative use of 
avatars. Any other suitable use may be implemented. 

[0094] Another feature that may be implemented in the 
virtual city application is the ability to display local televi- 
sion programming. In particular, television programming for 
a particular region may be made available to the user 
currently in the virtual representation of that region. For 
example, as the user is navigating around his or her virtual 
hometown, the user may view a local broadcast of a local 
baseball game. 

[0095] The virtual city application may also give the user 
travel-related options — in both the real sense as well as the 
virtual sense. For example, the user may be given the ability 
to make airline and hotel reservations. The virtual city 
application may also have a tour feature, whereby the user 
may take a virtual tour of a particular destination. For 
example, if the user wishes to visit New York, the virtual city 
application may take the user on a guided tour of New 
York's tourist attractions. The user may be taken to virtual 
representations of real museums, monuments, neighbor- 
hoods, shows, and whatever else the user may wish to see. 
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If desired, a lour guide may accompany the tour via, for 
example, the avatar feature and the audio capabilities of the 
present invention, or alternatively via text messages that 
may accompany the images. 

[0096] Tours may be based on place, interest, age, or any 
combination of these or any other suitable factors. For 
example, the user may wish to see a tour of the jazz scene 
in New Orleans. The virtual city application may be made to 
oblige this request and give the user a tour of the jazz clubs, 
and jazz museums and exhibitions in accordance with the 
present invention. This is merely an illustrative tour feature. 
Any such suitable tour feature may be used. 

[0097] In one embodiment of the present invention, arti- 
ficial intelligence may be used by the virtual city application 
to aid users in making full use of the features being offered. 
In its simplest form, artificial intelligence may take the form 
of a search engine. As was shown by search box 142 of FIG. 
4, users may simply enter keywords and have the virtual city 
application search for relevant items of interests in the 
virtual city images or for other types of suitable content. 

[0098] In another suitable approach, a back-end artificial 
intelligence component of the virtual city application may 
use a linear programming model to provide users with an 
intelligent personal assistant. FIG. 17 shows an illustrative 
linear programming model 260 upon which intelligent per- 
sonal assistant 264 may be predicated. A large number of 
variables 262 may be used to make the decision -making of 
linear programming model 260 more precise. Intelligent 
personal assistant 264 may be used to, for example, auto- 
matically perform personal chores over the Internet. More 
particularly, the chore management functionality of the 
present invention may be made programmable and autom- 
atable. 

[0099] By taking into account a user's preferences, habits, 
dislikes, location, schedule, and any other suitable criteria, 
the personal assistant feature of the virtual city application 
may perform many of the user's daily tasks without the need 
for user interaction. For example, searching for a service 
provider, such as a dentist may be based on the location 
criterion of the Upper West Side of Manhattan, the special- 
ization criterion of cosmetic dentistry, and the price criterion 
of moderately expensive. Any other suitable criteria may be 
used in having the intelligent personal assistant search for a 
suitable dentist. In another suitable embodiment, intelligent 
personal assistant 264 may be given the capability to rec- 
ommend restaurants based on preferences, pay the user's 
bills, manage finances, comparison shop for staple items, 
arrange convenient delivery times (e.g., depending on 
whether the user lives in a doorman building, when the user 
is home, etc.) and perform any other suitable services. Any 
such suitable capability may be given to intelligent personal 
assistant 264, 

[0100] In one suitable embodiment of the present inven- 
tion, the interface of the virtual city application with which 
the user interacts may be implemented using any suitable 
hardware, software, or both to create a three-dimensional 
world. For example, virtual reality markup language 
(VRML) may be used to create three-dimensional web 
pages. Any other suitable tools for creating three-dimen- 
sional graphics may be used. 

[0101] Three-dimensional representations of a city may be 
coupled with the use of a walkthrough interface. For 
example, as a user navigates around virtual city blocks, the 
movement appears as a smooth continuous movement. This 



type of movement is contrasted with the static walkthrough 
where, as previously described, the user may need to click 
on horizons or other suitable links to navigate to adjacent 
locations. Any such suitable type of movement may be used. 
If desired, the user may be given the option of which 
movement to use, based on, for example, connection speed, 
personal likes/dislikes, or based on any other suitable fac- 
tors. 

[0102] In one suitable embodiment, the user may be given 
the ability to use public transportation (or any other suitable 
form of transportation). For example, the user may take a 
virtual subway from one location in a virtual city to another 
location. This may be displayed to the user using a virtual 
representation of a subway and a virtual representation of a 
subway car. The user may virtually board the appropriate 
subway car required to travel to the desired location. This is 
merely an illustrative use of virtual transportation and the 
present invention is in no way limited to virtual subways. 
For example, there may be virtual taxis, virtual buses, virtual 
horses and carriages (e.g., to take through a virtual Central 
Park), virtual airplanes (e.g., to go from one virtual city to 
another), virtual cruises, or any other suitable form of virtual 
transportation. 

[0103] Whether using two-dimensional imaging or three- 
dimensional imaging, the amount of data that is communi- 
cated from data server 114 to user hardware 122 (FIG. 2) is 
potentially huge. In order to make the use of the virtual city 
application more efficient for those users whose connection 
speed is relatively slow, some data that is commonly com- 
municated from data server 114 to user hardware 122 may 
be made available to users on portable media, such as 
CD-ROM disks, DVD-ROM disk, minidisks, high-capacity 
floppy disks, or any other suitable type of portable media. 
The portable media may be sent to the user using, for 
example, the U.S. Postal Service or any other suitable mail 
carrier. The content of the portable media may include the 
most commonly accessed images of a particular region. 

[0104] In another suitable approach, there may be many 
different categories of data available on portable media. For 
example, one such category may relate to art museums. In 
such a case, the portable media may contain all of the virtual 
museum images of some of the most popular art museums 
in the world. The portable media may be required to be used 
in conjunction with the virtual city application. This is 
merely an illustrative example of the use of portable media. 
Any other suitable application for portable media may be 
used. 

[0105] In terms of revenue, the present invention may use 
any suitable revenue sources. FIG. 18 illustrates some of the 
possible sources of revenue of the virtual city service. For 
example, in one suitable approach, total revenue 256 may 
include advertisement revenue 250. For example, fees may 
be charged for embedded advertisement based on, for 
example, the amount of traffic for the particular virtual block 
on which the advertisement is to appear. In another suitable 
approach, fees may be based on the type of embedded 
advertisement that is to be used. For example, billboard 
advertisements may demand a higher fee than a small sign. 

[0106] Total revenue 256 may also include commission 
revenue 252. As was previously described, the virtual city 
application may process the payments for orders placed by 
users with merchants that are partnered with the virtual city 
service. In exchange, the virtual city service may take a fixed 
fee or a percentage of sale for services rendered in process- 
ing the payments. 
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[0107] Total revenue 256 may also include user fee rev- 
enue 254. The virtual city service may charge users access 
fees. Access fees may vary depending, for example, on the 
level of service desired. For example, a bronze membership 
may provide users with fewer features than a gold member- 
ship. The price for bronze membership may therefore be 
lower than the price for gold membership. In another suit- 
able approach, users may be charged on a pay-per-view 
basis. For example, if the user wishes to take a tour of a 
particular California vineyard, the user may be charged a fee 
to access that tour. Any such suitable features may be 
available on a pay-per-view basis. 

[0108] The preceding is merely illustrative of sources of 
revenue for the virtual city service. Any other such possible 
sources of revenue may be used to acquire revenue. 

[0109] Thus, methods and systems for presenting a virtual 
representation of a real city are provided. One skilled in the 
art will appreciate that the present invention can be practiced 
by other than the described embodiments, which are pre- 
sented for purposes of illustration and not of limitation, and 
the present invention is limited only by the claims which 
follow. 

What is claimed is: 

1. A method for centralizing localized e-commerce using 
a virtual city application that provides to users an interface, 
comprising: 

creating virtual representations of stores in the interface in 
which local merchants are allowed to sell goods and 
services; 

providing the users access to the virtual representations of 
stores, wherein the users are allowed to place items that 
are for sale in the virtual representations of stores into 
a single virtual shopping cart; and 

processing payment for the items in the virtual shopping 
cart using the virtual city application. 

2. The method defined in claim 1, further comprising 
creating virtual representations of stores in the interface in 
which web-based merchants are allowed to sell goods and 
services. 

3. The method defined in claim 1, wherein creating virtual 
representations of stores comprises digitizing images of 
storefronts and images of stores* interiors. 

4. The method defined in claim 1, wherein processing 
payment comprises: 

accessing payment information for the users from a 
database; 

receiving funds from the users in payment for the items in 
the virtual shopping cart based on the payment infor- 
mation; and 

placing an order for the items with the respective local 
merchants. 

5. The method defined in claim 4, further comprising: 

acquiring revenue by retaining a commission from the 
funds received; and 

paying the respective local merchants the funds received 
less the commission retained. 

6. The method defined in claim 1, wherein providing users 
access comprises: 

allowing the users to access the virtual city application via 
a communications network; 



displaying the virtual city application interface to the 
users; and 

enabling the users to navigate among virtual city block 
images viewable via the virtual city application inter- 
face. 

7. The method defined in claim 1, further comprising 
creating the virtual representations of stores for the respec- 
tive local merchants. 

8. The method defined in claim 1, further comprising 
acquiring revenue by charging the users a fee for use of the 
virtual city application. 

9. The method defined in claim 1, further comprising 
acquiring customer information from the users prior to the 
processing of payment, wherein the acquisition of the cus- 
tomer information occurs only once, and wherein the cus- 
tomer information is used for subsequent purchases made 
using the virtual city application. 

10. The method defined in claim 9 wherein the customer 
information comprises payment information. 

11. A system for centralizing localized e-commerce using 
a virtual city application that provides to users an interface, 
comprising: 

means for creating virtual representations of stores in the 
interface in which local merchants are allowed to sell 
goods and services; 

means for providing the users access to the virtual rep- 
resentations of stores, wherein the users are allowed to 
place items that are for sale in the virtual representa- 
tions of stores into a single virtual shopping cart; and 

means for processing payment for the items in the virtual 
shopping cart using the virtual city application. 

12. The system defined in claim 11, further comprising 
means for creating virtual representations of stores in the 
interface in which web -based merchants are allowed to sell 
goods and services. 

13. The system defined in claim 11, wherein means for 
creating virtual representations of stores comprises means 
for digitizing images of storefronts and images of stores' 
interiors. 

14. The system defined in claim 11, wherein means for 
processing payment comprises: 

means for accessing payment information for the users 
from a database; 

means for receiving funds from the users in payment for 
the items in the virtual shopping cart based on the 
payment information; and 

means for placing an order for the items with the respec- 
tive local merchants. 

15. The system defined in claim 14, further comprising: 

means for acquiring revenue by retaining a commission 
from the funds received; and 

means for paying the respective local merchants the funds 
received less the commission retained. 

16. The system defined in claim 11, wherein means for 
providing users access comprises: 

means for allowing the users to access the virtual city 
application via a communications network; 

means for displaying the virtual city application interface 
to the users; and 
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means for enabling the users to navigate among virtual 
city block images viewable via the virtual city appli- 
cation interface. 

17. The system defined in claim 11, further comprising 
means for creating the virtual representations of stores for 
the respective local merchants. 

18. The system defined in claim 11, further comprising 
means for acquiring revenue by charging the users a fee for 
use of the virtual city application. 

19. The system defined in claim 11, further comprising 
means for acquiring customer information from the users 
prior to the processing of payment, wherein the acquisition 
of the customer information occurs only once, and wherein 
the customer information is used for subsequent purchases 
made using the virtual city application, 

20. The system defined in claim 9 wherein the customer 
information comprises payment information. 

21. A system for centralizing localized e-commerce using 
a virtual city application that provides to users an interface, 
comprising: 

a central site at which virtual representations of stores are 
created for use in the interface in which local merchants 
are allowed to sell goods and services; 

a data server configured to provide the users access to the 
virtual representations of stores, wherein the users are 
allowed to place items that are for sale in the virtual 
representations of stores into a single virtual shopping 
cart; and 

hardware at the central site configured to process payment 
for the items in the virtual shopping cart using the 
virtual city application. 

22. The system defined in claim 21, further comprising 
hardware at the central site configured to create virtual 
representations of stores in the interface in which web-based 
merchants are allowed to sell goods and services. 

23. The system defined in claim 21, further comprising 
hardware at the central site configured to digitize images of 
storefronts and images of stores' interiors. 

24. The system defined in claim 21, further comprising: 

an information database that stores payment information 
for the users; 

a communications network that is used to transfer funds 
for payment for the items in the virtual shopping cart 
using the payment information; and 

hardware at the central site configured to place an order 
for the items with the respective local merchants. 

25. The system defined in claim 21, further comprising: 

a communications network configured to allow the users 
to access the virtual city application; 

graphics processing and display hardware configured to 
display the virtual city application interface to the 
users; and 

user control hardware configured to enable the users to 
navigate among virtual city block images viewable via 
the virtual city application interface. 

26. The system defined in claim 21, further comprising 
hardware at the central site configured to create the virtual 
representations of stores for the respective local merchants. 

27. The system defined in claim 21, further comprising 
hardware at the centra! site configured to acquire customer 
information from the users prior to the processing of pay- 



ment, wherein the acquisition of the customer information 
occurs only once, and wherein the customer information is 
used for subsequent purchases made using the virtual city 
application. 

28. The system defined in claim 27, wherein the customer 
information comprises payment information. 

29. A method for providing a user an interactive virtual 
representation of a city via a communications network, 
comprising: 

storing images of substantially most city blocks making 
up the city in a database; 

linking the images to one another in substantially the 
same way the corresponding city blocks are linked; and 

allowing the user to navigate among the city blocks. 

30. The method defined in claim 29, further comprising 
inserting embedded advertisements within the images. 

31. The method defined in claim 29, further comprising: 

allowing the user to select areas of the images; and 

performing an action upon the user selecting one of the 
areas. 

32. The method defined in claim 31, wherein performing 
an action comprises displaying additional information. 

33. The method defined in claim 29, further comprising: 

displaying storefronts in the images; 

allowing the user to select at least one of the storefronts; 
and 

displaying a virtual representation of the inside of a store 
corresponding to a selected storefront. 

34. The method defined in claim 29, wherein allowing the 
user to navigate comprises: 

allowing the user to select a particular direction of a first 
city block; and 

displaying a second city block that is linked to the first 
city block. 

35. The method defined in claim 29, further comprising: 

displaying an image of a map having selectable areas; 

allowing the user to select the selectable areas; and 

displaying a city block that corresponds to an area 
selected from the map. 

36. A system for providing a user an interactive virtual 
representation of a city via a communications network, 
comprising: 

a database in which images of substantially most city 
blocks making up the city are stored; 

hardware configured to link the images to one another in 
substantially the same way the corresponding city 
blocks are linked; and 

hardware configured to allow the user to navigate among 
the city blocks. 

37. The system defined in claim 36, further comprising 
graphics editing equipment that is used to insert embedded 
advertisements within the images. 

38. The system defined in claim 36, further comprising 
hardware configured to: 

allow the user to select areas of the images; and 

perform an action upon the user selecting one of the areas. 
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39. The system defined in claim 38, wherein the hardware 
is further configured to display additional information. 

40. The system defined in claim 36, further comprising 
hardware configured to: 

display storefronts in the images; 

allow the user to select at least one of the storefronts; and 

display a virtual representation of the inside of a store 
corresponding to a selected storefront. 

41. The system defined in claim 36, further comprising 
hardware configured to: 

allow the user to select a particular direction of a first city 
block; and 

display a second city block that is linked to the first city 
block. 

42. The system defined in claim 36, further comprising 
hardware configured to: 

display an image of a map having selectable areas; 



allow the user to select the selectable areas; and 

display a city block that corresponds to an area selected 
from the map. 

43. A method for using a linear programming model to 
provide automated services based on a plurality of variables, 
comprising: 

using the linear programming model to process a user's 
preferences, schedule, and requests; and 

performing actions based on the linear programming 
model over a communications network. 

44. The method defined in claim 43, wherein the perform- 
ing actions is selected from a group consisting of making 
appointments, placing orders for items, placing orders for 
services, scheduling deliveries, and any combination 
thereof. 

***** 
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(57) ABSTRACT 

For a world that has a portion of the world distant from the 
point of view of the user represented in only two dimensions 
as a video on a video screen, when an object on the video 
screen undergoes a trajectory that takes at least a portion of 
it to a location in the world that is not represented by the 
video but instead is a location in the world that is represented 
by computer graphics, in addition to being able to continue 
to see such an object when it is rendered as computer 
graphics in the computer graphics part of the world, i.e., 
popped out from the video, one is able to interact with such 
an object. Thus, an object which pops out from a video into 
the computer graphics part of the world may be "investi- 
gated" by a viewer of the world. For example, the user could 
enter a store which popped out of the video, and engage in 
virtual shopping therein. The particular store which is actu- 
ally entered may be customized on a per user basis, e.g., as 
a function of geography. When the proprietor of a virtual 
store, or his representative, e.g., electronic agent, detects the 
avatar of one or more persons in the vicinity of the avatar of 
a store for which such proprietor's store corresponds to the 
virtual store for such passing persons, a message may be 
transmitted to such passing persons. 
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THREE DIMENSIONAL VIRTUAL REALITY FEtoile in the center of which is the Arc de Triomphe, built 

ENHANCEMENT TECHNIQUES b Y Napoleon Bonaparte, and the Arc de Triomphe as it is 

passed by the bicyclist pops out of the video, the user could 

TECHNICAL FIELD click on the popped -out Arc de Triomphe which is now in 

, - 5 the computer graphics part of the wo rid and obtain historical 

TTiis invention relates to the integration of computer informa [ ion £ ou F t ilf displaye d in a window on the 

graphics and video to provide a realistic three dimensional Alternatively, if the user is represented in the world 

virtual reality experience. ^ aQ avalaf riding on a me user avatar, in response 

BACKGROUND OF THE INVENTION *° a PP r °priate commands, such as mouse clicks, could be 

10 made to dismount the bicycle and further investigate the Arc 

The display of a three dimensional world to a viewer de Triomphe, e.g., entering therein, ascending a set of stairs 

requires considerable computation power, and it is typically and looking out from one or more viewpoints. The user 

costly to develop the necessary highly detailed models could then return to his bicycle and continue his journey in 

required for doing so. In order to simplify the problem, a the world which has a portion of the world distant from the 

portion of the world that is in the distance may be repre- 15 point of view of the user represented in only two dimensions 

sented in only two dimensions as a video displayed on a asa video on the video screen and the remainder of the world 

surface, e.g., a screen. By video it is meant the common as computer graphics. 

usage of the term, such as the placing or projecting of ^ another example, the user could enter a store which 

predefined images on the surface, e.g., the electronic version popped out of the video, and engage in virtual shopping 

of filmed moving pictures. Thus, such a world is essentially 2Q therein. In accordance with an aspect of the invention, the 

truncated in length to the screen on which the video is particular store which is actually entered may be customized 

displayed. A great reduction in computation power and cost on a per useT -j^us, f or different users who are 

can be achieved by such an arrangement. traversing the same course and seeing the same 

In U.S. patent application Ser. No. (case Carraro-Ensor representations, e.g., an avatar, of a store or vendor cart that 

2-7) it was recognized that a limitation of such a world 25 popped out from the video, who the particular vendor is that 

occurs when an object within the field represented by the will serve the user and provide him with the virtual shopping 

video undergoes a trajectory that takes it to a location in the service may be different for different users. In accordance 

world that is not represented by the video but instead is a with an aspect of the invention, the particular vendor may be 

location in the foreground which is represented by computer selected as a function of the actual geographic location of the 

graphics, namely, any portion of the object that is no longer 30 user, or a particular location that is associated with the user, 

on the video screen, disappears. Therefore, when an object If two or more users are simultaneously exploring the same 

within the field represented by the video undergoes a tra- world together, e.g., two friends bicycling together, in accor- 

jectory that takes it, or a portion thereof, to a location in the dance with an aspect of the invention, each user, or different 

world that is not represented by the video but instead is a subsets of the users, may be served by a different vendor, 

location in the foreground which is represented by computer 35 e.g., as a function of specified parameter, such as their 

graphics, such an object or portion is made to continue to be individual geographic locations. Alternatively, even if there 

visible to the user and is represented at the foreground is more than one user, and such users would otherwise be 

location using computer graphic techniques, rather than served by different vendors, e.g., as a function of their 

video. Thus, the video object "pops" out of the video and respective locations, in accordance with an aspect of the 

becomes visible, e.g., in front of, or to the side of, the video 40 invention, such users could specify that a subset, or all of 

screen, rather than becoming invisible because it is no longer them, should be served by a single vendor, 

on the video screen. In accordance with another aspect of the invention, when 

n.,,,...^ „ IKnn:KrnAM the proprietor of a virtual store, or his representative, e.g., 

SUMMARY OF THE INVENTION electronic agent, detects the avatar of one or more persons 

We have recognized that, for a world that has a portion of 45 in the vicinity of, e.g., passing, the avatar of a store for which 

the world distant from the point of view of the user repre- such proprietor's store corresponds to the virtual store for 

sented in only two dimensions as a video on a video screen, such passing persons, a message, such as an advertisement, 

when an object on the video screen undergoes a trajectory or other communication, such as opening a communication 

that takes at least a portion of it to a location in the world that channel, e.g., a chat service or voice communication 

is not represented by the video but instead is a location in the 50 channel, may be transmitted to, or initiated with, such 

world that is represented by computer graphics, in addition passing persons. The communication may be general in 

to being able to continue to see such an object when it is nature or it may be customized as a function of information 

popped out from the video into the computer graphics part available regarding the passing persons. Advantageously, a 

of the world, it is advantageous for one to be able to interact feeling of community may be engendered in the virtual 

with such an object. Therefore, in accordance with the 55 environment 

principles of the invention, an object which pops out from a BRIEF DESCRIPTION OF THE DRAWING 
video into the computer graphics part of the world may be 

"investigated" by a viewer of the world. One way that this The file of this patent contains at least one drawing 

may be achieved is by receiving an indication of a selection executed in color. Copies of this patent with color drawing 

of such an object by a user and, in response thereto, 6 0 ( s ) wiU provided by the Patent and Trademark Office 

providing the user with additional information about the upon request and payment of the necessary fee. 

selected object. Another way that this may be achieved is by In the drawing: 

receiving an indication of a selection of such an object by a FIGS. 1-3 show an example of a video object "popping" 

user and, in response thereto, allowing the user to explore out of a video and so becoming visible in front of, or to the 

within the selected object. 65 side of, the video screen; 

For example, if the user is bicycling down the Champs FIG. 4 shows an exemplary process by which objects 

Elysees in Paris, France, as one approaches La Place de within the field represented by the video surface that, due to 
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their projected motion are to be in the three dimensional 
world, are so displayed using computer graphics techniques, 
e.g., the object "pops out" of the video surface; 

FIG. 5 shows another exemplary process by which objects 
within the field represented by the video surface that, due to 
their projected motion are to be in the three dimensional 
world, are so displayed using computer graphics techniques; 
and 

FIG. 6 shows steps of a process by which aspects of the 
invention may be implemented. 

DETAILED DESCRIPTION 

Before describing how an object which pops out from a 
video into the computer graphics part of the world may be 
"investigated" by a viewer of the world, in accordance with 
the principles of the invention, an example of a video object 
"popping" out of the video and becoming visible in front of, 
to the side of, above, or below the video screen, rather than 
simply becoming invisible because it is no longer on the 
video screen, is shown in FIGS. 1-3. For simplification of 
terminology purposes, a portion of an object may simply be 
referred to as an object, since any portion of an object may 
be considered an object in its own right. 

FIG. 1 shows world 101, which is the Champs Elysees in 
Paris, France, as one approaches La Place de FEtoile in the 
center of which is the Arc de Triomphe, built by Napoleon 
Bonaparte. World 101 is divided into two portions, video 
screen 103, on which is shown the current frame of a video 
and the remainder of the world 105, which is represented 
using computer graphics techniques, and is thus referred to 
herein as computer graphics part (CG Part) 105. The current 
frame of video being displayed on video screen 103 includes 
police van 113 and Arc de Triumph 115. Within CG Part 105 
there are various elements, such as bicyclist 107, represent- 
ing the user, road 109, and sky 111. 

Note that the viewpoint of the user is actually behind the 
representation of the user in the form of bicyclist 107, Also 
note that police van 113, which is part of the current video 
frame being shown on video screen 103, is moving slower 
than bicyclist 107, so that police van 113 will eventually be 
passed by bicyclist 107 as he continues to ride toward Arc 
de Triumph 115. 

FIG. 2 shows world 101 of FIG. 1, but at a later time. At 
the time of FIG. 2, the frame of video being shown on screen 
103 is from a view closer to Arc de Triumph 115. Such a 
frame may have resulted, for example, from moving the 
camera that captured the video closer to Arc de Triumph 115. 
As a result of the camera location when the frame of video 
on screen 103 that is shown in FIG. 2 was taken, only a 
portion of police van 113 was captured video frame. The rest 
of the police van 113 was out of view of the camera, and 
hence is not visible within the current frame of video on 
screen 103 that is shown in FIG. 2. However, from the 
viewpoint of the user, looking at world 101 as it appears in 
FIG. 2, the remaining portion of police van 113 should be 
visible, notwithstanding that it is no longer within the 
boundaries of video screen 103. Therefore, the remaining 
portion of police van 113 is displayed using computer 
graphic techniques within world 101 as element 201, which 
is apart of CG Part 105. 

FIG. 3 shows world 101 of FIG. 1, but at a time even later 
than that of FIG. 2. Thus, at the time of FIG. 3, the frame of 
video being shown on screen 103 is from a view still closer 
to Arc de Triumph 115 than that of FIG. 2. As a result of the 
camera location when the frame of video on screen 103 that 
is shown in FIG. 3 was taken, none of police van 113 is 
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visible within the current frame of video on screen 103 that 
is shown in FIG. 3. However, from the viewpoint of the user, 
looking at world 101 as it appears in FIG. 3, police van 113 
should be visible to the user's left, notwithstanding that it is 

5 no longer within the boundaries of video screen 103. 
Therefore, the entirety of police van 113 is displayed using 
computer graphic techniques within world 101 as element 
201, which is a part of CG Part 105. At least a portion of 
element 201 will continue to be displayed as part of CG Part 

1(J 105 until element 201 passes completely from the user's 
viewpoint. 

FIG. 4 shows an exemplary process by which objects 
within the field represented by the video surface that, due to 
their projected motion are to be displayed in the three 
dimensional world, e.g., in front of, to the side of, above, or 

35 below, the video surface, are so displayed using computer 
graphics techniques, so that the object "pops out" of the 
video surface. In typical embodiments of the invention, the 
objects to be displayed using computer graphics are 
predefined, and their location in the video and the time 

20 associated with its position within the video is known, so 
that the time and extent at which the object needs to be 
popped out of the video is known. However, the techniques 
of the invention may be employed with computers and 
software which are sufficiently sophisticated to track recog- 

25 nizable objects within the video surface and to develop 
computer graphics models from such video representations 
in order to create the popped out computer graphics. 

The process is entered in step 401 when the user selects 
a video to view and the first frame of the video is to be 

30 displayed to a user. In step 403, the video frame to be 
displayed is retrieved. This may be achieved by retrieving 
data representing the frame which is pointed to by a pointer. 
Such a frame pointer is set initially, e.g., in step 401, to point 
to the first frame of the video. Next, in step 405, it is 

35 determined if there are any objects with the current video 
frame which is to be displayed on the video screen for which 
popping out needs to be initiated. This may be achieved by 
employing additional information associated with the frame 
that describes any objects that need to be popped out. Such 

40 additional information may also include data necessary to 
render the object as well as motion data which indicates the 
path of the object. Alternatively, this step may be achieved 
by analyzing the content of the video frame, perhaps in 
conjunction with prior frames and future frames, to deter- 

45 mine data necessary to render the object as well as motion 
data which indicates the path of the object. In addition, the 
viewpoint given to the user by the overall display may be 
incorporated in the determination. Such a viewpoint is 
determined by camera controls, which may be set by the 

50 user. Of course, a combination of the foregoing techniques 
may also be employed. In step 407, a computer graphics 
display engine routine is initiated for each of the objects 
determined in step 405. Each of these computer graphics 
routines are supplied with the information necessary to 

55 generate their respective objects as time passes with each 
frame for display on the video surface. For example, the 
information supplied for each object may include the afore- 
mentioned motion data which indicates the path of the 
object. 

60 Next, steps 409 and 411 are performed substantially in 
parallel with step 413. The degree of parallelism to be 
achieved should enable the popped out objects to be dis- 
played substantially while the frame is displayed so that the 
user perceives them as being displayed together at the same 

65 time. 

In step 409, a location on the viewscreen is determined for 
each of objects for which a computer graphics display 
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engine routine is executing. In step 411, each of the objects A process for implementing this embodiment is shown, in 

for which a computer graphics display engine routine is flow chart form, in FIG. 5. Note that prior to entering the 

executing that are still within the view of the viewer are process shown in FIG. 5 a three-dimensional world includ- 

rendered. Optionally, the computer graphics display engine mg a ll the elements that must pop out of the video surface 

routine for any object that has moved so that it is no longer 5 must be pregenerated and stored in memory. Also, spatial 

within the view of the viewer may be deleted. However, such information which describes the location of the video screen 

computer graphics display engine routines should only be lhe three-dimensional world must be associated with 

deleted provided that it is known that the motion path of the frames of the video ^ stored ^ memory , 

object or the viewpoint of the viewer will not again bring „, , . A ™ , iL . A 

, J , . . . r to to The process is entered in step 501 when the user selects 

mat object into view. 10 a video to view and the first frame of the video is to be 

video frame is displayec I in step 413 Control then dis { d tQ a ^ , n 503 ^ video frame tQ ^ 

passes from both steps 411 and 413 to step 415. In step 415 displayed ^ retrieved> ^ may be achieved by retrieving 

the next video frame is advanced to, provided that there is data re mi the frame which is inted t0 b a inter . 

one. This may be achieved by incrementing or otherwise Sucfa a frame mter fe M ini(iaU in m {Q ^ 

updating the value of the frame pointer. Thereafter, control 15 tQ the first frame of the videQ m 505j lhe location of the 

passes back to step 403 and the process continues as videQ screeQ wilhjn me three-dimensional world is deter- 

desenbed above. mined. As noted, such information may be implied, e.g. if 

In another implementation of the basic process of FIG. 4 f the vidco is movmg at a constant rate of speed, or it 

the video screen is moving within the overall three- may be m separate information that is associated with the 

dimensional world being displayed to the user. Instead of 2 0 video frame. Next, in step 507, the visible portion of any pop 

determining which objects require the initiating of popping oul elements ^ determined as a function of the current video 

out based on information about objects in the frame being surface i ocat i on an a the user viewpoint. 

displayed on the video surface, as described above, the ™_ c , t - A _ , _ nft c , . . . „ 

. / ... .u . , r.u j Thereafter, steps 507 and 509 are performed substantially 

initiation may be based on the current location of the video „ . ~ J c „ r t . , . 

tl _ „ t . . , , j . • m parallel. The degree of parallelism to be achieved should 

screen within the overall three-dimensional world being 25 . . . • . . . , . . * .* n 

,. . , t . „ . . , . 6 " enable the popped out objects to be displayed substantially 

displayed to the user. For example, if the video screen is , t , e r rr . J .u .u 

. . ., . • . • ./ * * * . c while the frame is displayed, so that the user perceives them 

displaying a video which gives the user the feeling of asbeingdisp i ayedlogelheraUhesametime . Inste p 5 07,the 

traveling down the Champs Elysees each successive video deUrm ? ed H v J ble ^ of eacb of the out elements 

frame .s actually a representauon of the view from a location ^ ^ ^ frame ^ ^ m 

further and further down the street. As the frames are 30 P t . (U c , 4 . + eA 4 ♦ en 

, - ^ . Control then passes from both steps 507 and 509 to step 511. 

delayed and the user s viewpoint moves further down the , n 5U ^ ^ ^ ^ ^ iM 

street, ,t may be necessary for objects, e.g., a bus, . ^ ^ ^ Qne ^ be achievcd incrementi or 

pedestrian, or a parked car, to pop off the video screen and otherwise dati the va , ue of , he frame imer 

become represented by computer graphics The computer ^ back , 0 st 503 and me a 

graphics display engine routine for each of the objecte may 35 continU£s ^ described above 

be mitiated as a function of the distance traveled by the m „ . , 

video screen down the street, which, for example, may be , ^ foregoing .assumes that the video frame to be dis- 

eithcr indicated in additional information associated with the P la y ed 1S . *>}> w L mch u determines the location of the video 

frame, or, for simple situations, determined as a predeter- Wlthin the three dimensional worki However the 

mined function of the time that the video has been running. 40 co ° verse ma y als u 0 be implemented, i.e., the location of the 

T t . uj-.t.u- .* *u video screen within the three-dimensional world is set, and 

In another embodiment of the invention, a three- . . . . r . , .. . . ' , 

j. ! -11 1 c Aa . ■„,„,, • 1. iUa this determines which video frame is to be displayed, lo do 

dunensional world ready for rendering, mcludmg all the , eM . ..... • , c 4 u 

. .... . . c .u • j r • j 1 a so, in step 501 or 511, m which the video frame to be 

elements that must pop out of the video surface, is modeled ' . . K , . , * . , t , , . , . + . f 

^ . r r ,. . . 1 j * 1 j • r displayed next is determined, the desired location of the 

in memory. The three-dimensional world includes inform a- ' ' ... .... .. , • 

, J , . . . . j -. r u. video screen within the three-dimensional world is ascer- 

tion such as the spatial location, size and onentation tor each 45 , . . , , c c 

c , , . *l 4 * w . . r r tained. This location may be specified as a function of user 

of the elements that must pop out. Motion information for _ . . C . J r , . . , . 

the pop out elements may also be included. In this mpul.Forex^le,if Ihcs^ta^d^la^co^nds 

, r ,. r ..... . • -.u- .u 11 to the view of a user on a bicycle, the location of the video 

embodiment, the video screen is moving within the overall , , ... . 

. . t . . . . . j. , j . .u screen may be determined by sensors mounted to a station- 
three-dimensional world that is being displayed to the user, ,. / , . .j. - . , . ... 

. -u j u u c r .i_ j u *■ 1 ,« ary bicycle which detects riding of the bicycle by the user, 

as described above. Each frame of the video has spatial 50 . J J , £ , . c , ., & . r . . . . r .. 

. - . ... 1 t . r.u -j A search of the information identifying the location of the 

information which describes the location of the video screen c j.^j. - iur j 

•.u-.u.u 1 • . . ... cu frames is conducted to determine which frame corresponds 

within the three-dimensional world stored in memory. Such , . . , .■ c.. . .. c *. 

. r t . u * i- j ■ j* . 1 ■ j • to the current location of the screen, and the frame pointer 

information may be implied, e.g., if the video screen is • . . L . r 

moving at a constant rate of speed, or it may be in separate 15 set 10 P oinl 10 that frame ' 

information that is associated with the video frame. Using 55 ^ various method of determining which frame is dis- 

the information about each video pop out element as well as P la y ed ma y mc6 separately, or their usage may be 

the information about the location of the video screen allows intermixed, depending on the need of the application being 

the spatial location of each video pop out element, as well implemented. 

as the portion of each video pop out element that is not Note that frames for which it is known that no pop out 

blocked by the video screen and is within the viewpoint of 60 elements will be displayed need not be associated with 

the user, to be determined for each frame. Therefore, when spatial information which describes the location of the video 

the video screen is at a particular location, the portion of any screen within the three-dimensional world stored in 

video pop out element that is visible, e.g., in front of or to memory. The video is simply displayed on the video screen 

the side of, above, or below, the video screen may be and no pop out elements are rendered, 

rendered on the display. Thus, a match may be achieved 65 Any presto red information, such as frames, and informa- 

between what is seen on the video screen and the computer tion associated therewith, as well as models of the three - 

graphic pop out elements. dimensional world, may be stored in different segments. 



01/07/2004, EAST Version: 1.4.1 



US 6,2! 

7 

Such segments may be located at a single physical location, 
e.g., within a single computer, or at different physical 
locations, such as in various computers that are connected by 
a network, and made available for use with embodiments of 
the invention. Moreover, the elements which process the 
information to generate the display may, but need not be, 
contained within a single computer. 

One way that an object which pops out from a video into 
the computer graphics part of the world may be "investi- 
gated" by a viewer of the world, in accordance with the 
principles of the invention, is by receiving an indication of 
a selection of such an object by a user and, in response 
thereto, providing the user with additional information about 
the selected object. For example, additional information 
could be displayed on the display screen in response to the 
viewer selecting the object, such as with point and click 
selection system, the most common type of which are mice, 
trackballs, and touchpads. Alternatively, in response to the 
user selection, a new window could be overlayed on the 
video display containing additional information. Another 
way that this may be achieved is by receiving an indication 
of a selection of such an object by a user and, in response 
thereto, allowing the user to explore within the selected 
object. This may be done within the context of the world 
which is being explored or external thereto. One way this 
may be achieved is by having a uniform resource locator 
(URL) associated with the object in the CG part of the world. 
When the object is selected by the view, the viewer is taken 
to a page or file corresponding to the URL. Alternatively, the 
viewer may be jumped to another world model which 
corresponds to the object being further explored. 

For example, if the user is bicycling down the Champs 
Elysees in Paris, France, as described above, and, as the Arc 
de Triomphe is passed by the bicyclist it pops out of the 
video, the user could click on the popped-out Arc de 
Triomphe which is now in the computer graphics part of the 
world and obtain historical information about it, e.g., dis- 
played in a window on the screen. Alternatively, if the user 
is represented in the world as an avatar riding on a bicycle, 
the user avatar, in response to appropriate commands, such 
as mouse clicks, could be made to dismount the bicycle and 
further investigate the Arc de Triomphe, e.g., entering 
therein, ascending a set of stairs and looking out from one 
or more viewpoints. The user could then cause his avatar to 
return to the bicycle and continue the journey in the world 
which has a portion of the world distant from the point of 
view of the user represented in only two dimensions as a 
video on the video screen and the remainder of the world as 
computer graphics. The description of the interior of the Arc 
de Triomphe may have been directly associated with infor- 
mation employed for popping it out of the video or infor- 
mation pointing to the necessary description of the interior 
of the Arc de Triomphe, such as a pointer to another "world" 
or a URL, may have been directly associated with informa- 
tion employed for popping the Arc de Triomphe out of the 
video. 

As another example, the user could enter a store which 
popped out of the video, and engage in virtual shopping 
therein. In accordance with an aspect of the invention, the 
particular store which is actually entered may be customized 
on a per user basis. Thus, for different users who are 
traversing the same course and seeing the same 
representations, e.g., an avatar, of a store or vendor cart that 
popped out from the video, who is the particular vendor that 
will serve the user and provide him with the virtual shopping 
service may be different for different users. In accordance 
with an aspect of the invention, the particular vendor may be 
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selected as a function of the actual geographic location of the 
user, or a particular location that is associated with the user. 
If there is more than one user who are simultaneously 
exploring the same world together, e.g., two friends bicy- 

5 cling together, in accordance with an aspect of the invention, 
each friend may be served by a different vendor, e.g., as a 
function of their individual geographic locations. 
Alternatively, even if there is more than one user, and such 
users would otherwise be served by different vendors, e.g., 

30 as a function of their respective locations, in accordance 
with an aspect of the invention, such users could specify that 
a subset, or all of them, should be served by a single vendor. 

In accordance with another aspect of the invention, when 
the proprietor of a virtual store, or his representative, e.g., 

15 electronic agent, detects the avatar of one or more persons 
passing the avatar of a store for which such proprietor's store 
corresponds to the virtual store for such passing persons, a 
message, such as an advertisement, or other communication, 
such as opening a communication channel, e.g., a chat 

20 service or voice communication channel, may be transmitted 
to, or initiated with, such passing persons. The communi- 
cation may be general in nature or it may be customized as 
a function of information available regarding the passing 
persons. Advantageously, a feeling of community may be 

25 engendered in the virtual environment 

FIG. 6 shows steps of a process by which aspects of the 
invention may be implemented. The process of FIG, 6 is 
entered in step 601, which may be, for example, alter 
completion of execution of step 411 of FIG. 4 or step 507 of 

30 FIG. 5. Thereafter, in step 603, additional information which 
is necessary to support the permitted interaction between the 
viewer and the popped out object and is beyond what is 
necessary to simply generate the computer graphics visual 
model of the object is retrieved. Such information may be 

35 included along with the information associated with the 
video that is used to pop the object out of the video. In step 
605 any geographic functions associated with the object are 
resolved. For example, if the object which popped out of the 
video was a vendor's cart, such as may be found when 

40 bicycling around Central Park in New York City, the geo- 
graphic function may specify a restaurant of the type that 
serves the same kind of food that is available from vendor 
cart. Such food may include, for example, frankfurters, 
hamburgers, sodas and snacks. The type of restaurant that 

45 corresponds to the cart is then used along with the actual 
physical location in the real world of the user, e.g.. Omaha, 
Neb., to determine a local restaurant that serves food of the 
same type and provides delivery service. The information 
provided by exploring the cart is linked to information 

50 corresponding to the determined restaurant, e.g., available 
menu selections and prices therefor. The viewer may order 
an item from the restaurant by manipulating objects on the 
cart, and the restaurant could then deliver them directly to 
the viewer. Advantageously, users who are in different real 

55 world locations but are jointly negotiating a virtual world, 
may be supplied with substantially the same services, 
enhancing their experience in the virtual world. When there 
is more than one restaurant to which the object may be 
associated, the resolution of the object may also be made 

60 dependent upon knowledge about the user, such as may be 
stored in a user profile. 

Next, in step 607, a communication channel may be 
enabled between the popped out object and an entity, which 
may have been geographically resolved as described above. 

65 For example, if the popped out object is the above-described 
vendor cart, a communication channel may be opened to the 
corresponding restaurant, or an agent therefor. The restau- 
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rant may transmit a message, such as "Get your hot dogs and point of view of a viewer of said world, to a location 

cold soda here." The message may be manifest by having an that is off of said video screen, so that said object can 

avatar of a vendor behind the cart appear to speak the no longer be displayed on said video screen and will 

message, either using text-to-speech or by having a repre- disappear from said world; 

sentation of the speech transmitted directly. A so-called 5 changing the representation of said object from a video 

"chat" service may be used to enable bidirectional commu- representation to one modeled as computer graphics 

nication between the viewer and the vendor. whereby said object remains visible in said world, 

Another exemplary object to which a communication albeit not on said video screen, and said video screen 

channel may be opened is a billboard. Such billboards may continues to exist in said world; 

operate in one of several ways. For example, the billboard 1Q receiving an indication of a selection of said object by said 

when popped out may signal an advertiser of the approach viewer when said object's representation is modeled as 

of a potential customer and in response thereto the advertiser computer graphics" and 

may immediately transmit a message to the billboard in accessmg additional information about said selected 

accordance with an aspect of the invention. The particular object 

advertiser selected may be fixed worldwide for all viewers, j$ 2 ^ [nvMiQn as defined in daim x wnerein said object 

or it may be geographically resolved for each viewer. ^ a Pinboard 

Alternatively the billboard when popped out may signal an 3 ^ invention as defined in daim x whefein said object 

advertiser of the approach of a potential customer and enable ^ a stQre 

the advertiser to transmit a message to the billboard should 4 ^ * invenlion as defined in claim x wherein said o5ject 

the advertiser wish to do so, e.g., after having reviewed iq ^ a vendor sland 

information which may be sent to it about the viewer, in 5 ^ invenlion as defined in daim x wherein said object 

accordance with an aspect of the invention. ^ a g ara g e SQ \ G 

The process exits in step 609 and then the processes of 6 ^ mvent ion as defined in claim 1 wherein said 

FIGS. 4 or 5 may be resumed. additional information is determined as a function of speci- 

The process of FIG. 6 need not be limited to occurring 25 fied parameters of said viewer of said world, 

within the time span of one frame. 7. The invention as defined in claim 1 wherein said 

Those of ordinary skill in the art will recognize that any additional information is determined as a function of a 

simulation of a joint exploration of a world may employ the location in the real world of said viewer of said world, 

principles of the invention. For example, users exploring a 8. The invention as defined in claim 1 wherein said 

world may take together automobile rides, boat rides, plane 30 additional information is determined as independent of a 

rides, train rides, and even spaceship rides. A rowboat ride location in the real world of said viewer of said world, 

may be arranged for users having rowing machines which 9. The invention as defined in claim 1 wherein said 

are coordinated with images of the world seen by the users. additional information is determined as a function of pref- 

Alternatively, the users may be walking together through the erences of said viewer of said world, 

world, e.g., each user walks on his own treadmill which is 35 10. The invention as defined in claim 1 wherein said 

coordinated with images of the world seen by the users. object is a store selected from among a plurality of stores as 

Similarly, users can ski together, e.g., each is on a skiing a function of a location of said viewer and said additional 

simulator machine, such as a Nordictrak, which is coordi- information is relevant to objects for sale at said store, 

nated with the images of the world seen by the users. In 11. The invention as defined in claim 1 wherein said 

addition, other forms of commercial transactions may be 40 object is a store selected from among a plurality of stores as 

engaged in besides shopping. For example, users motoring a function of specified parameters of said viewer and said 

down a virtual highway together may encounter a toll booth additional information is relevant to objects for sale at said 

which pops out of the screen, and at least one of the users store. 

must pay the toll before their exploration of the world is 12. The invention as defined in claim 1 further including 

allowed to proceed. Similarly, users may approach an attrac- 45 the step of opening a communication channel between said 

tion such as a virtual amusement park or a virtual safari. viewer of said world and a location defined as a function of 

Ticket booths which appear on the video screen may pop out said additional information. 

from the video screen and the users must each obtain, e.g., 13. The invention as defined in claim 1 wherein said 

buy, or have bought for them, a ticket in order to proceed additional information is determined as a function of a 

into the attraction. 50 location in the real world of said viewer of said world and 

The foregoing merely illustrates the principles of the further including the step of opening a communication 

inventions. It will thus be appreciated that those skilled in channel between a viewer of said world and a location 

the art will be able to .devise various arrangements which, defined as another function of said additional information, 

although not explicitly described or shown herein, embody 14. A method for use by a plurality of users in exploring 

the principles of the invention and are included within its 55 a three dimensional world in which a first portion of said 

spirit and scope. world is modeled as computer graphics and a second portion 

What is claimed is: of said world is represented by a video, wherein said video 

1. A method for use in accessing an object in a three is made up of a sequence of images displayed on a video 

dimensional world in which a first portion of said world is screen which exists in said first portion of said world, said 

modeled as computer graphics and a second portion of said 60 video displaying a plurality of objects, the method compris- 

world is represented by a video, wherein said video is made ing the steps of: 

up of a sequence of images displayed on a video screen determining that an object which is represented as part of 

which exists in said first portion of said world, said video said video on said video screen has moved, from the 

displaying a plurality of objects, the method comprising the point of view of at least one of said users, to a location 

steps of: 65 that is off of said video screen, so that said object can 

determining that an object which is represented as part of no longer be displayed on said video screen and will 

said video on said video screen has moved, from the disappear from said world; 
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changing the representation of said object from a video 22. The invention as defined in 17 wherein said commu- 

representation to one modeled as computer graphics nicaiion channel is presented as a advertisement avatar in 

whereby said object remains visible in said world, said first portion of said world. 

albeit not on said video screen, and said video screen 23- The invention as defined in 17 wherein said commu- 

continues to exist in said world; 5 nication channel is a voice channel. 

• • • ■ c ... e .. . , 24. The invention as defined in 17 wherein said commu- 

receiving an indication of a selection of said object by a . , , ulu w 

°. ■ j • * « i • nication channel is a data channel, 

user when said object s representation is modeled as 25 ^ „ defined m 17 wnerein ^ commu . 

computer graphics; and nicatioQ channd fc a cfaat sessioQ 

transmitting to at least at least two distinct subsets of said 26. A method for use in accessing an object in a three 

plurality of users additional information about said 10 dimensional world in which a first portion of said world is 

selected object; modeled as computer graphics and a second portion of said 

wherein said additional information about said selected world is represented by a video, wherein said video is made 

object that is supplied one of said at least two distinct U P of a sequence of images displayed on a video screen 

subsets of said plurality of users is different from the which exisls in said first portion of said world, said video 

additional information about said selected object that is 15 displaying a plurality of objects, said world being explored 

supplied to another of said at least two distinct subsets b V a P luralu V of lhe method comprising the steps of: 

of said plurality of users determining that an object which was represented as part 

15. The invention as defined in claim 14 wherein said of said video on said video screen has moved, from the 

object is an avatar of a store and wherein said additional , n P° ,n i. of f v,ew of f user of said w ° rld > * a J™" lhat 

. r i- j . j c -j . i -i- . 20 is oil of said video screen, so that said object can no 

■nfonnanon supplied to said one of said at least two distinct , be d; , d Qn ^ vide0 ^ an(J wm 

subsets of said plurality of users relates to a first real world ^ fr( J ^ 

store and wherein said additional information supphed to . . tU * *■ ** j u- . c _ %j 

... - . , . ... • i c * j changing the representation of said object from a video 

said other one of said at least two distinct subsets of said representation to one modeled as computer graphics 

plurality of users relates to a second real world store, said l$ whereby ^ object femains yisible m said wodd> 

first and second stores being different. albeh not on said video screeilj and ^ video screen 

16. The invention as defined in claim 15 wherein said continues to exist in said world; and 

additional information actually supplied to said one of said opening a plurality of communication channels when an 
at least two distinct subsets of said plurality of users and to avatar of said user ^ a specified distance of said 
said other of said at least two distinct subsets of said 3Q object in said first portion of said world when said 
plurality of users is determined as a function of defined object's representation is modeled as computer graph- 
parameters regarding said one and said other distinct sub- KS 

sets - 27. The invention as defined in claim 26 wherein a first of 

17. A method for use in accessing an object in a three said commuruca tion channels connects said user and an 
dimensional world in which a first portion of said world is 35 actor on beha]f of a first real world item repr esented by said 
modeled as computer graphics and a second portion of said object and a second of said commun i ca tion channels con- 
world is represented by a video, wherein said video is made necls said user and an actor on benalf of a second real world 
up of a sequence of images displayed on a video screen Ucm rcprescnted by said object, said first and second actors 
which exists in said first portion of said world, said video being different. 

displaying a plurality of object, the method comprising the 4Q 28 ^ invention as defined m claim 2 6 wherein said a 

steps of: communication channel of said plurality of communi- 

determining that an object which was represented as part ca tion channels connects to a first billboard avatar in said 

of said video on said video screen has moved, from the world and carries thereto, for display thereon, a first message 

point of view of a user of said world, to a location that an d a second communication channel of said plurality of 

is off of said video screen, so that said object can no 45 communication channels connects to a second billboard 

longer be displayed on said video screen and will avatar j n sa j d wor ] d and carr ies thereto, for display thereon, 

disappear from said world; a second message, said first message being different from 

changing the representation of said object from a video said second message. 

representation to one modeled as computer graphics 29. The invention as defined in claim 26 wherein a first 

whereby said object remains visible in said world, 50 communication channel of said plurality of communication 

albeit not on said video screen, and said video screen channels connects said user and an actor on behalf of said 

continues to exist in said world; and object and a second communication channel of said plurality 

opening a communication channel when an avatar of said of communication channels connects a second user and 

user is within a specified distance of said object in said another actor on behalf of said object, said first actor being 

first portion of said world when said object's represen- 55 different from said second actor and each of said first and 

tation is modeled as computer graphics. second actors representing different respective real world 

18. The invention as defined in claim 17 wherein said items that are each represented in said world by said object, 
communication channel connects to a display avatar in said 30. A method for use in processing a view of a three 
world. dimensional world in which a first portion of said world is 

19. The invention as defined in claim 17 wherein said 60 modeled as computer graphics and a second portion of said 
communication channel connects to a billboard avatar in world is represented by a video, wherein said video is made 
said world. up of a sequence of images displayed on a video screen 

20. The invention as defined in claim 17 wherein said which exists in said first portion of said world, said video 
communication channel connects said user and an actor on displaying a plurality of objects, the method comprising the 
behalf of said object. 65 steps of: 

21. The invention as defined in claim 20 wherein said determining that an object which is represented as part of 
actor is an autonomous agent representing said object. said video on said video screen has moved, from the 
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point of view of a viewer of said world, to a location 
that is off of said video screen, so that said object can 
no longer be displayed on said video screen and will 
disappear from said world; 

revealing a representation of said object as computer 5 
graphics whereby said object remains visible in said 
world, albeit not on said video screen, and said video 
screen continues to exist in said world; 

accessing additional information about said object not 
required to display said object as said computer graph- 10 
ics to said viewer having a viewpoint external to said 
object when said object's representation is modeled as 
computer graphics. 

31. The invention as defined in claim 30 further including 
the step of prestoring said representation of said object as 15 
computer graphics. 

32. The invention as defined in claim 30 further including 
the step of displaying said additional information to said 
viewer. 

33. The invention as defined in claim 30 further including 20 
the step of displaying said additional information to said 
viewer in a window distinct from said computer graphics. 

34. The invention as defined in claim 30 wherein said 
video displayed an exterior view of said object and wherein 
said additional information includes information for gener- 25 
ating an interior view of said object. 

35. A method for use in processing a view of a three 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second portion of said 
world is represented by a video displayed on a video screen 30 
which exists in said first portion of said world, wherein said 
video is made up of a sequence of images, the method 
comprising the steps of: 

determining that an object which is represented as part of 35 
said video on said video screen has moved, from the 
point of view of a viewer of said world, to a location 
that is off of said video screen, so that said object can 
no longer be displayed on said video screen and will 
disappear from said world; 4Q 

revealing a representation of said object as computer 
graphics whereby said object remains visible in said 
world, albeit not on said video screen, and said video 
screen continues to exist in said world; 

arranging a data structure for said object as computer 45 
graphics so that said viewer can access other informa- 
tion about said object other than a view of its repre- 
sentation as computer graphics which was revealed as 
it moved to said location modeled as computer graphics 
when said object's representation is modeled as com- 50 
puter graphics. 

36. A method for use in accessing an object in a three 
dimensional world in which a first portion of said world is 
modeled as computer graphics and a second portion of said 
world is represented by a video, wherein said video is made 55 
up of a sequence of images displayed on a video screen 
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which exists in said first portion of said world, said video 
displaying a plurality of objects, the method comprising the 
steps of: 

determining that an object which was represented as part 
of said video on said video screen has moved, from the 
point of view of a viewer of said world, to a location 
that is off" of said video screen, so that said object can 
no longer be displayed on said video screen and will 
disappear from said world; 
changing the representation of said object from a video 
representation to one modeled as computer graphics 
whereby said object remains visible in said world, 
albeit not on said video screen, and said video screen 
continues to exist in said world; and 
arranging a data structure for said object as computer 
graphics so that said viewer can access other informa- 
tion about said object other than a view of its repre- 
sentation as computer graphics to which it was changed 
when said object represented by said video moved to 
said location modeled as computer graphics. 

37. The invention as defined in claim 36 further including 
the steps of: 

receiving an indication of a selection of said object by said 
viewer; and 

accessing said additional information about said selected 
object. 

38. The invention as defined in claim 37 wherein said 
additional information is displayed in a separate window. 

39. The invention as defined in claim 37 wherein said 
additional information defines an interior view of said 
object. 

40. The invention as defined in claim 37 wherein said 
additional information is used to associate said object with 
a real world object. 

41. The invention as defined in claim 37 wherein said 
addtional information is used to associate said object with a 
real world object as a function of a real world location of 
said viewer of said object. 

42. The invention as defined in claim 37 wherein said 
additional information is used to associate said object with 
a real world object as a function of preferences of said 
viewer of said object. 

43. The invention as defined in claim 37 wherein said 
additional information is used to establish a communication 
channel between said object and a real world object as a 
function of a real world location of said viewer of said 
object. 

44. The invention as defined in claim 37 wherein said 
additional information is used to establish a communication 
channel between said object and a real world object and 
wherein said real world object is selected as a function of a 
real world location of said viewer of said object. 

***** 
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INTERACTIVE ELECTRONIC SHOPPING shopping facility into encoded digital signals representing 

SYSTEM AND METHOD tDe images as would be seen by a shopper at a physical 

embodiment of the shopping facility; means for storing the 

This application is a continuation of copending applica- <- nc ° ded digital signals in a computer data base; means for 

tion Ser. No. 08/838,087 filed on Apr. 15, 1997. 5 displaying a video representation of the shopping facility in 

response to the stored encoded digital signals; and control 

BACKGROUND OF THE INVENTION means for the shopper to control the means for displaying 

such that the means for displaying causes the displayed 

This invention relates generally to interactive electronic video image to change at the shopper's command to corre- 

shopping systems and methods and more particularly, but spond to what the shopper would see were the shopper to 

not by way of limitation, to such systems and methods move through a physical embodiment of the shopping 

utilizing the Internet so that an individual shopper can access facility. 

shopping information from his or her home or office, for In a particular implementation, the interactive electronic 

example. shopping system comprises: a digital camera to digitize a 

Commercial use of the Internet, and particularly the 15 shopping facility into digital signals representing images of 

World Wide Web, presently allows some degree of investi- the shopping facility; a central computer data base to store 

gation and ordering of products offered for sale. The con- the digital signals; a communication link connected to the 

venicnce that this can offer to a shopper would be enhanced central computer data base; a local computer connected to 

with regard to at least some types of shopping if the shopper the communication link; and a control interface connected to 

could see a virtual reproduction of an actual store and the 2 o lhe local com P uter 10 enable a shopper to control the local 

products offered there as they are actuaUy displayed at the computer such that the local computer obtains data from the 

store. Convenience would also be enhanced if the shopper central computer data base in response to the digital signals 

could examine the displayed products in a manner similar to stored therein and displays video images of the shopping 

how one would at the actual store. facility in response to the obtained data. 

Further convenience would be added by creating one or 25 The present invention also provides a method of creating 
more lists of what the shopper may need to buy based upon a virtual shopping facility for interactive shoppmg by corn- 
historical or predetermined ordering patterns or upon actual puter. This method comprises: selecting a phystcally embod- 
selections at the time the shopper is moving through the ied shopping facility in which a shopper can purchase from 
virtual store shown on the shopper's display. among a plurality of "products; .converting images of at least 

^ , . . - - a i« a a' *» ii * i 30 a portion of the shoppmg facility and the plurality of 

Thus, there is the need for a video-based digitally impte- ^ encoded digital signals; storing the encoded 

mented interactive electronic shoppmg system which allows P ^ a * ^ m ^ m; md {6[ng 

a customer to vww the contents of a particular shopping ^ ^ encoded sucfa ^ 

facility in a format that provides the feeling of shopping for tions Qf ^ ^ encoded d - UaJ s - ls afe aDd 

the items in that particular facility. Use of this should be lfansmissible tQ a ter for displaying , on a monitor of 

available at the customer s home or business through a computer , virtual movement within a visual representa- 

convenient device, such as a personal computer, television Qf ^ f and for permitting examination 

or other consumer type electronic visual display device. ^ ^ represenlalions of products in response t0 

SUMMARY OF THE INVENTION ^ cicd stored encoded di & ilal si & nals - . 

40 , The present invention also provides a computer- 

The present invention provides a novel and improved implemented method of simulating movement of a shopper 

interactive electronic shopping system and method that through a shopping facility to enable an individual to 

satisfy the aforementioned needs for making shopping more observe, inspect and select a product in the shopping facility 

convenient to the shopper. Using the present invention, a through operation of a computer. The method comprises: 

shopper can browse through a virtual duplicate of an actual 45 selecting a shopping facility having a known configuration 

store in a manner similar to being in the actual store itself. 0 f pro ducts displayed at the selected shopping facility such 

The shopper can examine individual products, and select or mal a shopper can move among the displayed products to 

not select ones for purchase. Through the selection process, observe, inspect and select from among the displayed prod- 

one or more types of lists can be created (e.g., alphabetical ucts; simulating the configuration of displayed products in a 

or by location in the store). Historical lists based on past 5Q computer, including translating the known configuration of 

ordering and predetermined buying frequency can also be displayed products into encoded configuration-defining 

provided. Special displays or information can also be pro- electrical signals in memory of a first computer and using the 

vided to alert the shopper to specials on particular products. electrical signals to create in a second computer video 

Products can be located through a directory that correlates representations of the configuration of displayed products; 

all the products with their respective locations in the store. 55 indicating to at least the second computer a first position of 

Changes at the actual store can be implemented in the virtual a shopper in the shopping facility and displaying through a 

store. monitor of the second computer the video representation of 

This invention benefits the marketer of the products the configuration of displayed products at the first position; 
because, for example, it allows for customer loyalty to be and indicating to at least the second computer a continuous 
developed since the exact depiction of the marketer's store 60 change from the first position to a second position and 
actually enables the customer to become very familiar with displaying through the monitor of the second computer a 
the store and its particular products and to stay abreast of changing video representation of the configuration of dis- 
changes. It also enables a customer to reduce the actual played products corresponding to a moving scene the shop- 
shopping time and to ease or enhance the shopping experi- per would see in moving from the first position to the second 
ence. 65 position in the shopping facility. 

An interactive electronic shopping system of the present The present invention still further provides a computer- 
invention comprises: means for converting images of a implemented method of making a shopping list for purchas- 
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ing products from a shopping facility, comprising perform- group of items. By "mouse" double click or touch of an 

ing each of the following steps using a computer: accessing indicator on the hot spot, for example, a portion of the screen 

a predetermined data base of encoded electrical signals displays additional information concerning the item, 

representing a shopping facility and products available for The additional information concerning an item includes 

purchase from the shopping facility; retrieving from the data 5 any information desired. Examples are product size, price, 

base a sequence of the encoded electrical signals and dis- unit price, last purchase date, location in the store, additional 

playing moving images of at least a portion of the shopping product advertisement including audio information, infor- 

facility and products to simulate movement through the mation concerning other sizes or related products available 

portion of the shopping facility in response to the retrieved an d a «yes" or "no" to order or place an item on a list. From 

sequence of encoded electrical signals; and selecting dis- 10 the running list of items selected for purchase, additional 

played products and making a list of the selected displayed lists can be automatically created. One such list could 

products. include a list of items ordered with the cost of each, the 

Therefore, from the foregoing, it is a general object of the running total cost, and store location. A list arranged in order 

present invention to provide a novel and improved interac- of store location could be selected for use in speeding time 

tive electronic shopping system and method. Other and 35 required for obtaining the items while in the actual store. If 

further objects, features and advantages of the present inven- the order is to be filled by another party, the list could order 

tion will be readily apparent to those skilled in the art when the items for selecting from a warehouse or other storage 

the following description of the preferred embodiments is location. Items which must be weighed or selected on a 

read in conjunction with the accompanying drawings. weight basis could be estimated based upon prior purchase 

20 history. Unique items such as flowers or vegetables could be 

BRIEF DESCRIPTION OF THE DRAWINGS ordered by category, number, and total cost or amount 

FIG. 1 is a block diagram of the preferred embodiment required (if applicable). 

f f . %cnnt • ° * Additional lists can be automatically available at the 

system of the present invention. ... • i . • . ™ 

„ . , , , „ . , . . beginning of the interactive virtual shopping session. These 

FIG. 2 is a block diagram of a particular implementation 25 mdude epared Usts of items routinely purchascd , 

of the preferred embodiment system of the present inven- ilems purchased in the pasl> and items ^ for reorder 5ased 

tl0n - on prior purchase history. 

FIG. 3 is a representation of an illustrative screen display Qne purpose of this invention is to allow persons that 

at a local computer of the particular implementation of FIG. caDnot Qf do not want tQ visit a store t0 ^ an order 

2- 30 electronically to a store where personnel will select the items 

FIG, 4 is a representation of another illustrative screen and either deliver them or hold them for pick up by the 

display at the local computer represented in FIG. 3. customer. Automatic electronic payment capability is pref- 

FIGS. 5 and 6 are representations of alternatives for the erably also available as a part of the order process, 

screen display of FIG. 4. 35 The present invention enables pre -shopping of a particular 

FIG. 7 is a representation of still another illustrative store in the comfort of one's home, office or any location 

screen display at the local computer represented in FIG. 3. where a suitable electronic interface, (e.g., a computer 

FIG. 8 is a representation of a further illustrative screen ha ™S Internet access capability) is available. TTie informa- 

display at the local computer represented in FIG. 3. tion can be searched, selected and lists prepared for use 

_ . _ ... c . ... t 4 . A . while in the store, thereby reducing the shopping time. It 

FIG 9 is a representation of a still further illustrative 40 ^ unfa ^ Har w ^ th e particular ™ ston> 

screen display at the local computer represented in FIG. 3. ^ desifes Qr unfamiUar ^ lhe store t0 efficiemly do the 

FIGS. 10A-10B show a flow chart of part of the method actual shopping using the lists produced, 

of the present invention. invention nas particular application to any provider 

DETAILED DESCRIPTION OF THE 45 of g ocx3s or services where customers, clients or patients 

INVENTION typically go to an actual shop, office or other physical 

facility. It is especially applicable to any merchandiser of 
Overview consumer products including grocery stores, mass merchan- 
A digital camera provides an inexpensive and versatile disers of consumer goods such as variety or department 
way for a store owner to electronically capture images of the 50 stores, drug stores and other similar stores selling many 
store's contents for computerized interactive viewing by its items. Another specific application would allow a person to 
customers. The present invention uses this to enable a have the visual experience of walking into a restaurant, 
viewer of a display screen to visually proceed through a sitting down at a table, opening up a menu and ordering as 
virtual embodiment of the store in a fashion similar to if the person were in the restaurant. The order could be 
walking up and down aisles at the actual store. The viewer 55 electronically transmitted for preparation for "carry out" or 
can view items on each aisle with the ability to move closer "delivery." The restaurant could create a video image of 
to read images of actual labels on products on shelves or in each of its menu items for display upon activation of a 
displays. The digitization of the images generates electroni- particular "hot spot" on the viewed menu. Unless otherwise 
cally stored data that allows for efficient substitution of new stated or limited by a particular stated context, the terms 
images if products are moved or changed in a store. Items 60 "shopping facility" and "store" as used in this specification 
that the store chooses to highlight, such as for "sale" or and the accompanying claims encompass all such places to 
"special" items, can flash or otherwise be made distinctive which the present invention is applicable. The term "shop- 
as the customer approaches them on the virtual journey per" refers to a user of the present invention, 
through the displayed store. Preferred Embodiments of System and Method 

"Hot spots" are imposed upon the display screen, such as 65 An electronic shopping system in accordance with the 

at the location of the store price label for a particular product present invention is represented in FIG. 1. The system 

or at a logical location before a display of one particular creates an electronically produced, electronically transmis- 
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sible visual replica of a display of goods or other objects at are used to give the shopper at the display means 12 the 

a shopping facility or other particular environment. The sense of moving through the aisles of products in the grocery 

shopping facility is conveyed to a shopper through the store 2 example. 

present invention as a specific shopping environment to be ^ 6[ { mcms n ^ programmed t0 also incIude 

met when the shopper is actually at the facility This expe- 5 means f()r ammunm\m% information about the selected 

dues the shoppei : moving through the actual facility because q{ tQ ^ sfa ^ information 

the shopper has famihanty with it. Any type of environment f . . . . .y „ . . e 

• j * * j i_ ; (L /• «• II u includes ordering information as well as other information 

may be depicted; however, the present invention will be ... , b . . . ™ . f 

specifically described with reference to a conventional gro- 35 ^ ^f™ apparent below. He information commum- 

cery store 2 having conventional aisles stocked with typical in " ,ed * «fi»™««on that has been previously entered into the 

products 4. It is to be noted, however, that other items, such 10 da,a ba * of '".f , stor ^ e ^ ™ ™ e f * f °™ all ° n ra ° e 

as business cards, product brochures, business information, audlb Jy through speakers forming part of the 

, . , ~ • t . . . display means 12 or visibly through the aforementioned 

machines, personnel, meetmg or examination rooms, etc., *\/ r =f . _ 

can be depicted monitor or other display device of the display means 12. 

eleclronic shopping system of the present invention 15 Jhc display means 12 also includes means for generating 

includes means 6 for converting images of the shopping and Splaying one or more Tists of products ordered. These 

facility into encoded digital signals representing the images hsts caD * P™"" 10 alphabetical order or in a sequence 

as would be seen by the shopper at the physical embodiment according to the locations of the ordered products in the 

of the shopping facility. The image converting means 6 can popping facihty so that the shopper can go lo the actual 

be used to provide any desired view that will produce a 20 hc ^V and efficiently and expeditiously move through the 

desired video replica of the shopping facility, particularly the faciht y t0 obtain the ltems 10 be Purchased. The lists can be 

interior envirooment and products of the facility. For organized in any other suitable manner as well, 

example, the image converting means 6 can be used to Another type of list that can be provided by the means for 

generate electrical signals from which these features can be generating and displaying a list of the present invention is a 

depicted at a distance for a broader view or close up to 25 list of products previously ordered by the shopper and the 

provide sufficient clarity to read significant product labeling. date each listed product was last ordered by the shopper. 

A particular implementation of the image converting This type of list can also indicate the frequency with which 

means 6 is a digital camera 8 shown in FIG. 2. This can be each listed product has been ordered or is to be ordered by 

a conventional known type of product that generates, in *e shopper. One implementation of this is to program the 

response to the received light emitted or reflected from the 30 display means 12 so that actual ordering or selecting of 

shopping facility features, digital signals representing these products is automatically tracked and a frequency compu- 

detected images of the shopping facility. The image con- tation made, such as either on the basis of the time since last 

verting means 6 can be implemented in other ways, such as ordering the same product or an average time based on a 

by a scanner device using a pictorial representation of the historical record of the orders for the same product. These 

shopping facility or by a computer operator using a known 35 lists are generated and displayed by suitable programming 

type of graphics program to create images on a computer means within the display means 12 or as obtained from the 

(e.g., the computer operator could draw an image based on storage means 10. 

a picture of an actual location or based on an artist's The display means 12 can also display selected images 

rendering, or as a fictitious place if actual shopping by the that are visually distinctive relative to the video represen- 

user is not to occur). It is preferred, however, to use a digital 40 tations of other images to highlight these images to the 

camera as illustrated in FIG, 2 so that a substantially exact shopper at the display means 12. For example, certain 

representation of the actual facility can be quickly obtained products that are on "special" at the grocery store 2 might 

and ultimately displayed in the electronic shopping system flash on the display screen of the display means 12. 

of the present invention. Known types of photographic (e.g., Alternatively, there might be a text indication noting that 

telescopic lensing) or digital imaging techniques can be used 45 such product is on special. Such distinctive display can be 

in creating long range or close up views with desired detail. applied for other purposes, examples of which include goods 

The electric signals from the image conversion means 6 that may need to be reordered, items that have been selected 

are stored in a suitable storage means 10 preferably forming for current purchase, etc. This feature of distinctive display 

part of a computer data base. The storage means 10 can be is implemented by means for modifying the respective 

implemented in any suitable manner, such as by one central 50 encoded digital signals of the selected images so that the 

memory or multiple distributed memories, or individual video representations of these selected images displayed by 

memories (e.g., diskettes or CD-ROMs provided to the the display means 12 are visually distinctive to the video 

shoppers for their individual use at home or business). The representations of other images not represented by modified 

storage means 10 of whatever type is used within the system encoded digital signals. The modification can be imple- 

such that data can be removed and replaced as changes are 55 mented by the tagging or linking technique referred to below 

made in the actual facility 2. whereby the programming commands which cause the 

The data from the storage means 10 is used to provide a ima ge to flash or the "special" notification text to appear arc 

video representation of the shopping facility as presented l inke d with the respective signals defining the respective 

through a display means 12 as represented in FIG. 1. The product image. 

display means 12 includes a video monitor of any suitable 60 The electronic shopping system shown in FIG. 1 also 

type, such as a computer monitor or a television screen, in includes control means 14 for the shopper to control the 

conjunction with other conventional hardware and software display means 12 such that the display means 12 causes the 

that allow the downloading and use of the data from the displayed video image to change at the shopper's command 

storage means 10 in the display means 12. The information to correspond to what the shopper would see were the 

downloaded can be downloaded all at one time or incre- 65 shopper to move through the physical embodiment of the 

mentally as specifically required. Visible images can be shopping facility 2. Ill is includes a suitable interface device 
displayed either as still or moving images. Moving images such as a mouse, keyboard, light pen, or touch screen 
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function that allows the shopper to indicate to the display readily usable by the shopper, and the computer 18 can be 

means 12 that other video representations are to be displayed of this type or larger depending upon data handling capa- 

or that a product is to be more closely examined or ordered, bility required for a specific implementation. The printers 16 

for example. An example of this is to have the control means shown in FIG. 2 are of any desired conventional known type 

14 allow the shopper to interact with the display of one aisle 5 compatible with the respective computer 20 (e.g., laser, ink 

of the grocery store 2 with which a shopping cart image is jet, etc.). 

also displayed. The control means 14 allows the shopper to The communication link connecting the data storage 

cause the display means 12 to move the shopping cart down computer 18 with the local computers 20 can include a 

the aisle as if the shopper were pushing it. The control means telephone line, cellular link, cable television or other con- 

14 further allows the shopper to cause the display means 12 10 nection medium, but in general the local computers 20 will 

to stop at a desired part of the aisle, turn toward that be remote from the data storage computer 18. As mentioned, 

particular part, and zoom in on a particular product. The the particular implementation of FIG. 2 uses the Internet 22 

control means 14 can also allow the shopper to cause the as the communication link. The computer 18 and the com- 

display means 12 to show movement of the selected product puters 20 have suitable known programs to interface with 

off the shelf for closer examination or for moving the 15 the Internet and communicate over it to achieve the data 

displayed representation of the selected product into the acquisitions and transfers referred to in this specification, 

represented shopping cart in the same manner as the shopper The programming of the computer 18 and the computers 

would actually function in the real grocery store 2. The 20 is, in general, known conventional programming which 

shopping cart representation is a specific embodiment of a stores data, accesses data, communicates data, manipulates 

selected product accumulation space. Another embodiment 20 data, and displays data in manners known in the art. One 

of such a space is a counter in a clothing store where a cash significant aspect of the data handling is, however, that the 

register is located and where the shopper would collect digitized signals from the digital camera 8 need to be stored 

selected clothing items in moving them from the racks of m the data storage computer 18 in a manner which allows 

clothing to purchase them. This also provides means for particular portions of data associated with respective images 

ordering the selected products in that the display means 12 25 displayed through the local computer 20 to be identified and 

can record the selections and add them to an order list. The independently manipulated. One way to do this is to create 

control means 14 can provide for eliciting textual, graphical links or identifiers with the digitized video data so that it can 

or audible information about the product (e.g., pull -down be located and associated with other information. For 

menus or windows containing the information). example, identifiers can be spatial coordinates of the interior 

The electronic shopping system shown in FIG. 1 also 30 of the grocery store 2. As the data is recorded by the digital 

includes printing means 16 from which one or more selected camera 8 and stored in the computer 18, the data of a 

lists can be printed. This allows the shopper to take the particular portion of the store is associated in the data 

printed list and go to the actual grocery store 2 (or other storage computer 18 memory with a respective x-y-z coor- 

represented facility) to perform the actual shopping. The dinate. Other identifiers could be manually assigned or 

present invention also contemplates, however, electronically 35 otherwise associated by a programmer looking at an image 

ordering the selected item so that the facility 2 receives the on the screen of the local computer 20 and clicking on areas 

electronic order, fills it and has the selected items ready for of products and assigning identifiers to those areas. In 

pick up by or delivery to the shopper as desired. whatever manner is chosen, this allows a particular product 

A particular implementation of the system shown in FIG. or other visual segment shown on the screen of a local 

lis illustrated in FIG. 2. A representation of one actual aisle 40 computer 20 to have a unique identifier to which other 

of actual products 4 is digitized using the digital camera 8 information can be associated. For example, this other 

referred to above. The digital signals output from the digital information can be a set of encoded signals which cause the 

camera 8 are provided to a data storage computer 18 respective product to flash when it is displayed on the local 

embodying the storage means 10 of FIG, 1. computer 20 to indicate that that product is being offered at 

The display means 12 and the control means 4 of FIG. 1 45 a special price or has been ordered or needs to be reordered, 

are implemented in FIG. 2 at least in part by one or more This also allows product information, selector "hot spots" 

local computers 20, A communication link, illustrated in and other features to be associated with respective products. 

FIG. 2 as the Internet 22, connects the data storage computer This allows, for example, a hierarchy of classifications to be 

18 with the local computers 20. The data storage computer assigned to each product (e.g., "canned goods," "green 

18, the one or more local computers 20 and the Internet 22 50 beans," "(specific brand name)") to facilitate text-based 

are used to embody the storage means 10, the display means searching for particular products or types of products. This 

12 and the control means 14 of the FIG. 1 embodiment. still further allows for discrete or segmented replacement of 

Particular functions and structures of the system can be data; for example, when a certain product or group of 

implemented in either the data storage computer 18 or the products is changed in the actual store, a corresponding 

local computer 20 as desired; however, in general, the main 55 change can be made to the stored signals in the data storage 

data base of digitized signals provided from the digital computer 18 by replacing or modifying only that data at the 

camera 8 is contained in the central data storage computer 18 corresponding coordinates in the data base, 

and the control means 14 is part of each local computer 20 With the data entered, linked or tagged as described 

for the implementation of FIG. 2. The display occurs at the above, and formatted with appropriate associated 

local computer 20, but particular programming to accom- 60 information, the shopper at one of the local computers 20 

plish the functions of the present invention can reside in accesses the data storage computer 18 through the local 

either the computer 18 or the computer 20 as desired. The computer and communication link. In the particular 

computers 18 and 20 are of any suitable known type having implementation, this involves accessing a home page for the 

enough memory and operating speed to handle the amount grocery store 2 that is also stored in the data storage 

of data used in defining the shopping facility and the 65 computer 18. To perform this access, the local computer 20 

operations performed therewith. Preferably each computer is turned on, the Internet is accessed by conventional Inter- 

20 is a personal computer type or other comparable type net access provider, and the address of the home page is 
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entered (e.g., http://www.abcgrocery.com). A home page 24 
such as illustrated in FIG. 3 is then displayed, for example. 

From the home page 24 illustrated in FIG. 3, the shopper 
can "go shopping" or "make lists" or use a "product locator." 
These are function selectors designed into the home page 
which connect the shopper to features of the present inven- 
tion illustrated in FIGS. 10A-10B. 

Referring to FIGS. 10A and 10B, a shopper that "goes 
shopping" from the home page 24 shown in FIG. 3 can travel 
through the virtual store, look at specific products, request 
more information about products, select products for 
purchase, and make lists. The shopper can also enter order 
information and print one or more lists. 

In traveling through the store, the shopper sees the interior 
of the virtual embodiment of the store as displayed through 
the respective local computer 20. At the beginning of a travel 
through the store, the display can show entry through the 
front door of the store, selection of a shopping cart, and 
movement to a conventional starting point; however, pro- 
gramming can be used to allow the customer to start at any 
selected point within the environment of the shopping 
facility 2. Additionally, programming can allow the shopper 
to travel down a main aisle and look down each stocked aisle 
to see images of the products and/or menus of products 
contained in that aisle. The shopper can then go down each 
aisle as desired. The beginning of one such stocked aisle is 
shown in FIG. 4. FIG, 5 shows another type of display of the 
aisle including an index 26 of the items in the store, a list 28 
of items on "special," and a list 30 of items to be ordered 
based on a predetermined or a calculated frequency. FIG. 6 
shows still another type of aisle display, this one including 
a menu 32 of products available on that aisle. 

In general, traveling through the store involves the pro- 
gramming of one or more of the computers 18, 20 to 
determine the location of the shopper, access data, and 
display the retrieved data. This involves data transfers 
between the computer 18 and the respective computer 20 
whereby the desired data to be displayed is provided to the 
local computer 20. One way to determine the location of the 
shopper in the store is via a cursor 34 displayed on the screen 
as shown in FIG. 4, This location of the cursor 34 is oriented 
relative to the computer program defined three-dimensional 
space coordinate system set up for the shopping facility and 
providing one means of identifying the digitally recorded 
images. Once the shopper enters the shopping facility, the 
computer 18 or 20 keeps track of where the cursor 34 is 
located within the virtual three-dimensional space and the 
computer 18 or causes the associated data within a prede- 
termined region of that location to be displayed, again such 
as illustrated in FIG. 4 (and FIGS. 5 and 6). That is, the 
computer programming can be defined with preset spatial 
parameters that define the range of x-y-z coordinates to be 
shown at any given cursor location. This can be a fixed 
range, or different ranges depending upon the various posi- 
tions within each aisle, for example, or as otherwise desired. 

The present invention also allows for rotational orienta- 
tion so that if the shopper is in an aisle, looking in a 
particular direction, the computer 20 displays that view. 
Such rotational orientation can be implemented using the 
control means 14 such as specifically implemented by the 
arrow keys on a keyboard or by a joy stick. Thus, as the 
shopper travels down the aisle from the end shown in FIG. 
4, different data is retrieved from the computer 18 (or from 
data previously downloaded to the computer 20) to show 
movement along the aisle as indicated by the scene in FIG. 
7. Also shown in FIG. 7 is one technique for identifying to 
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the shopper items to be reordered and on special. Means 
other than the textual indicators of FIG. 7 can be used; for 
example, the products can be made to flash or can be 
highlighted in a predetermined color. 

5 When the shopper stops at a particular section to look at 
a product, as indicated by left, right, forward or reverse 
motion such as input through the arrow keys of the 
keyboard, a more limited, closer view of a section of the 
aisle can be viewed, such as shown in FIG. 8. Although not 

10 shown in the representation of FIG. 8 due to space and 
lettering constraints, the individual products, such as product 
36, appear on the display means as replicas of the actual 
products and include their labeling in sufficient definition 
that the labels can be read in at least close-up views. 

35 Associated with each product or group of like products is the 
corresponding price and information label 38 also in detail 
for reading in at least close-up views. 

An even closer view is shown in FIG. 9 in association 
with further information about a particular product and lists 

20 generated when products are selected. In FIG. 9, for 
example, each of the shelf labels 38 is a "hot spot" which 
causes a product information window 40 to be displayed 
when the cursor 34 is placed on the shelf label "hot spot" and 
clicked on. As shown in FIG. 9, the information can be 

25 visibly shown as in window 40; however, audible informa- 
tion can also be provided if the computer 20 has a speaker 
and the data from the computer 18 data base contains data 
for generating audio output. As another example, the product 
may be graphically presented in response to actuating the 

30 "hot spot"; for example, different discrete views of the 
product can be sequentially shown in maximum close-up 
views with sufficient resolution and clarity to allow the 
shopper to read product labeling. These are examples of 
implementations of the means for communicating informa- 

35 tion about a selected product and the means by which that 
selected product is identified. 

To obtain data by which the different views shown in 
FIGS. 4-9 can be provided, the digital camera 8 can be 
moved throughout the store to capture all of the images and 

40 this raw data stored in the computer 18. The computer 
programming is set up to define specific image ranges within 
each location of the cursor 34 so that under shopper control 
of the control means 14, the cursor location within the store 
can be noted and the images within the appropriate range of 

45 that location displayed. As the cursor 34 is moved and other 
control information given to the computer 18 or 20, move- 
ment along an aisle and zoom in and zoom out views can be 
shown. Alternatively, discrete segments of the store can be 
stored in the memory and these discrete segments pulled up 

so and displayed as the cursor 34 reaches appropriate locations 
within the three-dimensional coordinate system set up for 
the interior of the shopping facility. 

As the shopper moves through the virtual store displayed 
via the computer 20, products can be selected for purchase 

55 such as by entering data through the product information 
window 40 as shown in FIG. 9 (i.e., by selecting "yes" after 
"purchase") or by single or double clicking on the product 
or the "hot spot" in a distinctive manner from that used for 
calling up the product information. A selection can then be 

60 automatically added to one or more lists, examples of which 
are shown in FIG. 9. For example, the lists can be alpha- 
betical (42), by store location (44), or by warehouse location 
(or other "order fill" location) (46). At least a list such as list 
30 also automatically lists the number of items ordered, the 

65 unit cost and/or the total for each product, and a running total 
for the entire list. The programming for the computer 18 or 
20 can also allow the shopper to add notes to any of the lists 



01/07/2004, EAST Version: 1.4.1 



US 6,381,583 Bl 

11 12 

(e.g., an instruction for the person filling the order regarding The foregoing defines the preferred embodiment of a 

what to do if a particular selection is not available); entering computer-implemented method of simulating movement of 

a note can be by any suitable means of the system, such as a shopper through a shopping facility to enable the indi- 

by entering text through the keyboard of the computer 20. vidual to observe, inspect and select a product in the 

Once a list has been created for the products to be 5 shopping facility through operation of a computer. Stated 

purchased, the products can be ordered electronically another way, this is a method of creating a virtual shopping 

through the Internet system with instructions for delivery by facility for interactive shopping by computer, 

the store or for pickup by the shopper going to the store. Based on the system described above, this method com- 

Payment provisions (preferably secure ones) can also be prises selecting a physically embodied shopping facility 

included in the system. 30 having displays of goods physically disposed therein for the 

If the shopper wants to go to the store and go through the particular implementation of a grocery store, for example, 

aisles to select the individual items himself or herself, the The selected shopping facility has a known configuration via 

desired list can be printed (e.g., by selecting the "print" its ac tual construction and arrangement of products dis- 

button on any of lists 42, 44, 46) and taken to the store by played at the selected shopping facility such that a shopper 

the shopper. ' 15 ca n move among the displayed products to observe, inspect 

Returning to the home page 24 shown in FIG. 3, one or and from amon S the Splayed products, 

more list forms is displayed at the respective computer 20 if The method simulates the configuration of the displayed 

the shopper selects "make list." These can be of the same products in a computer, such as one of the computers 18 or 

type shown in FIGS. 5 and 9 and can allow the shopper 20. This includes in the illustrated implementation translat- 

eithcr to create a current list or to install a reorder list. 20 ing the known configuration of displayed products into 

Referring to FIG. 10A, to make a current list, the shopper encoded configuration-defining electric signals in memory 
enters the current date, selects any reorder items from an of the computer 18 and using the electric signals to create 
existing reorder list, enters new items to be selected and then vldeo representations of the configuration of displayed prod- 
controls the computer 20 to automatically organize in the 2S ucts thf0U &h the computer 20 (in another non-limiting 
manner the shopper wants (e.g., by location in the store or example, the signals can be stored on a storage medium used 
by price or alphabetically as illustrated in FIG. 9) and print directly in the computer 20). Conversion of the images of the 
the list if desired. The shopper can then end the session or physically disposed displays of goods into encoded digital 
"go shopping" to see if additional items are needed. sl S nals 15 b y means of the dl gi tal camera 8 for the Particular 

To make a reorder list, the shopper enters the current date, 30 of FIG ^ ™ e encoded di f' al si S nals fr ° m 

the item to be reordered and information regarding the the camera f are s ! ored ^ the computer 18 in this imple- 

f~^.^~, ™a c*™* *u„ r~~ r A»r Ko* ,\, mentation. Access is provided to these stored signals such 

reorder frequency, and stores the reorder list in memory in , . * . . , , -i , ° . ... 

•*u *u t 1ft t . * 10 m. i* * u that portions of the signals are selectable and transmissible 

either the computer 20 or the computer 18. The list can be . iL * j • i • * ^ . 

• . i j *u l. *u « u » a *u. to the computer 20 for displaying, on a monitor of that 

printed, and the shopper can then "go shopping or end the . *; J . & ' . f , 

r .j.j A i e n * 7 *u j computer, virtual movement along the display of goods 

session as desired. An example of a resultant output based on 35 * , 4 j . j J... . ( f • , ? (U 

... t n*,- »_ * 1-1^ e u i',in ul u represented by selected stored encoded digital signals. In the 

this type of list is shown in rlG. 5 by list 30, which shows „. r A .. . , T . . 

, . j , c iU 4 a * c I* 1* » in illustrated implementation this uses known Internet commu- 

ltems to be reordered as of the current date. Such a list 30 * 

. u j i j nication technology for accessing and transferring the req- 

can be defined by the foregoing to have a predetermined & & "i 

uisite siizoais 

order frequency or a list can be automatically maintained & 

based on actual ordering history (e.g., each time an actual 40 ^ sho PP er ming the system and method of the presenl 

order list is made, the historical list updates and determines invention is located at a respective one of the computers 20 

ordering frequency such as by computing an average time and indicates to that computer 20 a first virtual position of 

between purchases of the same product). the shopper within the displayed shopping facility. This first 

TUc third option shown on the home page 24 of FIG. 3 is P ositi ™ * indi ^ d b V the sh °PP e [ controlling the control 

to use the "product locator" Details of this are shown in 45 means ^ such as a ™ ouse ' 0 V™*' cursor f 3 t 4 at a 
t-t^ in*; a . jj- ♦ fn*u a * desired location within the displayed video representation of 
FIG. 10A. A predetermined directory of all the products . „ . _ Jt . f . 
avaUable from the store 2 is encoded in digital signals stored the configuration of displayed products, 
in the computer data base of the system such that this ^ shopper indicates to the computer a continuous 
directory is searchable by one or more words entered into the chan g e ih& position to a second position. This can 
computer 20 by the shopper (or other selection technique, 50 be done b y movin g the 34 alon S tne displayed image, 
e.g., an alphabetical List the shopper scrolls through). The In response the stored data within the predetermined display 
programmed computer 20 with its keyboard (for example) raQ g e of the cursor location is acquired to continuously 
provide means for enabling the shopper to search the defined chan g e tne Splayed images resulting in displaying through 
searchable directory in that the shopper types in one or more the monitor of the computer 20 a changing video represen- 
words to identify the product to be located (e.g., "green 55 tation of the configuration of displayed products correspond- 
beans"). The system then matches this to the stored directory in 6 t0 a movi og scene the shopper would see in moving from 
listing and retrieves and displays back to the shopper lhe first position to the second position in the actual shop- 
through the screen of the computer 20 where the product is pi°S facility. 

located in the store 2. This can be by a text response (e.g., The method of the present invention also includes dis- 

"aisle 2") or by displaying the video image of that portion of 60 tinctively displaying predetermined ones of displayed prod- 

the store as would be seen if the shopper moved among the ucts to indicate to the shopper that the distinctively dis- 

virtual aisles. The shopper can select these found items, played products are offered on a special basis or need to be 

make one or more lists, and order if desired, all as indicated reordered or are on the current order list, for example, 

in the flow chart of FIG. 10A. The shopper then can print the The method also includes indicating to the computer the 

hst(s), "go shopping," or return to the home page 24, also as 65 shopper's selection of a specific product. This can be by, for 

shown in FIG. 10A. Other specific implementations can, of example, clicking on the displayed product, or clicking on 

course, be used. the displayed product and dragging it to a video represen- 
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tation of a shopping cart, or answering a selection query in 
a display window on the screen. Selection can also be by a 
light pen, screen touch, keyboard actuation or other means. 
When a product is selected, it is automatically added to the 
selected product list. 

The invention also includes maintaining in the computer 
a historical list of products selected by the shopper. 

From the product selections, a list is created in the 
computer 18 or 20, which list contains the products to be 
purchased by the shopper at the actual selected shopping 
facility. The list can be organized in a sequential order in 
which the selected products are to be picked up at the actual 
selected shopping facility or at another location such as a 
warehouse where electronically transmitted orders are filled. 
Other list organizations can be used. 

When the list has been completed, it can be printed for the 
shopper to use when the shopper goes to the actual selected 
shopping facility, for example. 

As mentioned above, the hardware and software for 
implementing the system and method described above are in 
general conventional and known in the art. To the extent any 
modifications or additional hardware or software is needed 
in specifically implementing the invention, such modifica- 
tions and additions are within the skill in the art given the 
explanation of the invention set forth above. 

Thus, the present invention is well adapted to carry out the 
objects and attain the ends and advantages mentioned above 
as well as those inherent therein. While preferred embodi- 
ments of the invention have been described for the purpose 
of this disclosure, changes in the construction and arrange- 
ment of parts and the performance of steps can be made by 
those skilled in the art, which changes are encompassed 
within the spirit of this invention as defined by the appended 
claims. 

What is claimed is: 

1. A food vending method, comprising: 

maintaining an actual operating food vendor at a physical 

location, the food vendor having a customer service 

area; 

converting at the customer service area actual images of 
the customer service area and food served therein into 
encoded digital signals; 
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storing the encoded digital signals in a computer storage 
medium; and 

providing for remote computer communication link 
access to the stored encoded digital signals such that 
transmissions over the communication link responsive 
to the stored encoded digital signals display on a 
monitor of a computer at a location remote from the 
physical location of the food vendor a continuous 
motion visual representation of movement and food 
within the customer service area enabling a person 
viewing the monitor to have a visual experience of 
walking in the customer service area of the actual 
operating food vendor, sitting down at a table, and 
opening a menu as if the person were in the physical 
location of the actual operating food vendor. 

2. A method as defined in claim 1, wherein: 
converting actual images of the customer service area 

includes converting an actual image of a menu used in 
the customer service area; and 
the continuous motion visual representation includes the 
actual image of the menu. 

3. A method as defined in claim 2, wherein the food 
represented in the continuous motion visual representation 
appears in response to selecting at the computer the respec- 
tive food item from the actual image of the menu in the 
continuous motion visual representation. 

4. A method as defined in claim 2, further comprising 
transmitting an order for food from the computer to the food 
vendor over the communication link. 

5. A method as defined in claim 3, further comprising 
transmitting an order for food from the computer to the food 
vendor over the communication link. 

6. A method as defined in claim 3, wherein the selected 
food representation appears in continuous motion showing 
the selected food being served at the table. 

7. A method as defined in claim 1, wherein the continuous 
motion visual representation further enables the person 
viewing the monitor to have a further visual experience of 
seeing a food item being served to the displayed table after 
the person has selected the food item from the opened menu. 
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